big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
25,216 Commits 2 Branches 2 Tags
Commit Graph

9316 Commits

Author SHA1 Message Date
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9dd12b874ced1798f0c3ad0c160cb465)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb8139a08b92b5a94adfde544de978137c)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b071fe5b943542827fd8a496b137fa9)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
Harunobu Daikoku
db8af483d1 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:47:16 +09:00
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf429d28561dd9c582f5c55451213d89a4)
 2021-03-25 13:25:17 +00:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aad1b5a131d366aa5395f2904f0f6ecd)
 2021-03-25 05:09:11 +08:00
Takanobu Asanuma
9a0c5d0ca2 HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina. 
(cherry picked from commit 1fdfaebd98d9b8f40f616d03fa2f3b6b5c71b11b)
 2021-03-24 17:44:29 +08:00
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1bf4f0a45571c9fc4361e4e8a05f7ee)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
e4217be749 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 
(cherry picked from commit 5ab1db9e218e11d00035e234344ef90228d8898d)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
580634882e HDFS-14546. Document block placement policies. Contributed by Amithsha. 
(cherry picked from commit 17ffcab5f621400bd8bb47dc8fb29365f6e24ebd)
 2021-03-24 17:44:29 +08:00
Karthik Palanisamy
c96821ece8 HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 
(cherry picked from commit 735e85a1d2036585075b81d1fb2fd302e89eb363)
 2021-03-24 17:44:29 +08:00
Takanobu Asanuma
9e1e89b1b9 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:53:19 +09:00
sunlisheng
a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944e628345109b43cba2cd4d49ca8cc6b)
 2021-03-24 14:04:47 +08:00
Ayush Saxena
94766fdb13 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 
(cherry picked from commit ff84a57483320abf81bf941b81d13d5bee14084b)
 2021-03-24 14:04:47 +08:00
imbajin
4f79df4da1 HDFS-15551. Tiny Improve for DeadNode detector (#2265) 
Contributed by imbajin.

Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 89428f142fe7cee17bd1a0f5f207b6952ec79d32)
 2021-03-24 14:04:47 +08:00
sunlisheng
3ecd02ac78 HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 
(cherry picked from commit f8769e0f4b917d9fda8ff7a9fddb4d755d246a1e)
 2021-03-24 14:04:47 +08:00
Narges Shadab
f07bde90c9 HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 
(cherry picked from commit 115623a6ee4eb7391c7c26e1728fa153c0698803)
 2021-03-24 14:04:47 +08:00
Narges Shadab
3a355282c9 HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe90e3bb2ee783f956432db11e655b9e3)
 2021-03-24 14:04:47 +08:00
Narges Shadab
4abe6e6ac1 HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0604484ec3c48ac2acb32ea71a53d9a)
 2021-03-24 14:04:47 +08:00
Gautham B A
a216ee3b85 HDFS-15910. Improve security with explicit_bzero (#2793) 
(cherry picked from commit 714427c41401eacf3cab7c087d9e5d61a658716b)
 2021-03-24 14:04:47 +08:00
Toshihiro Suzuki
e9b06d942c HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 
(cherry picked from commit c12ddbd1de40b32bbe2f6a3e484abf843d6d92ae)
 2021-03-24 14:04:47 +08:00
Viraj Jasani
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2796) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:51:21 -07:00
Ayush Saxena
76c40a52d7 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:13:52 +05:30
Ayush Saxena
b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f669b2e2a6654cfacff22a090c462effc)
 2021-03-23 15:34:48 +00:00
Ayush Saxena
968c95bfba HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit ae13a5ccbea10fe86481adbbff574c528e03c7f6)
 2021-03-23 15:34:36 +00:00
Ayush Saxena
d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1d48407edf437131783b653dbc87bc1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena
793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 12:35:11 +00:00
litao
bc2bdee76c HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 
(cherry picked from commit 343ce8a8c955dcf64ad453e441947e8761247a5b)
 2021-03-22 20:30:53 +09:00
Ayush Saxena
0b43c65816 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad2396f8b42d60031001a25d9d6fa32e)
 2021-03-19 18:50:31 +00:00
Ayush Saxena
9665a0be73 HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:18 +05:30
Viraj Jasani
17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma
aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:15:53 +09:00
touchida
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 12:01:15 +09:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:07:48 +08:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:00:25 -08:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:35:45 +09:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)
 2021-03-01 11:50:23 +00:00
Surendra Singh Lilhore
a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
 2021-02-26 10:49:16 +08:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5d0024ab052429ef2f0c0b20e1ea258)
 2021-02-24 11:04:52 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2fa83ad4119954b32a3a8daac9d08134)
 2021-02-22 18:14:18 +08:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8bf96cf5a1ec75134fb67fdeb955e5b6)
 2021-02-22 15:24:37 +08:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a978e1b5220a961813d0e0c834010553)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Akira Ajisaka
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b173381204bd8d2c984f19c52104a419322)
 2021-02-16 14:52:31 +09:00