big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,381 Commits 2 Branches 2 Tags 264 MiB
29414871bd
Commit Graph

9005 Commits

Author SHA1 Message Date
litao
29414871bd
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2928) 2021-04-21 10:41:13 +09:00
hemanthboyina
90f63cb12c HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 
(cherry picked from commit 16a20503ca)
 2021-04-20 16:21:57 +01:00
Ayush Saxena
c93a2f6789 HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 
(cherry picked from commit c84e6beada)
 2021-04-20 12:13:39 +01:00
litao
dc830cf277 HDFS-15970. Print network topology on the web (#2896) 
(cherry picked from commit 85a3532849)
 2021-04-19 21:22:47 +09:00
Hui Fei
4c0b833bab HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 
(cherry picked from commit 0d42104119)
 2021-04-16 14:07:15 +09:00
Masatake Iwasaki
a3405cb56a HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f85)
 2021-04-15 12:39:07 +00:00
Wei-Chiu Chuang
bfba6f1f3c
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-12 23:55:32 -07:00
Stephen O'Donnell
0e6cd352b5
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. (#2813) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2021-04-09 10:25:14 +01:00
Viraj Jasani
4994b73eeb HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 
(cherry picked from commit 56bd968fb4)
 2021-04-09 11:13:39 +09:00
He Xiaoqiao
df99ac0399 HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df41997)
 2021-04-08 13:15:37 +01:00
Stephen O'Donnell
ef95f7a963 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7)
 2021-04-08 12:15:31 +01:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e6892675)
 2021-04-08 17:20:08 +08:00
Gautham B A
d7b31acbba
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 36014b8282)
 2021-04-06 18:23:38 +09:00
Viraj Jasani
8b4b3d6fe6 HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b)
 2021-04-06 14:18:31 +09:00
Viraj Jasani
2b5fd341b9 HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:20:56 +05:30
litao
7a7e194228 HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9e2d5d6d05)
 2021-04-05 17:49:27 +09:00
Gautham B A
e4d04a640b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74)
 2021-04-05 11:42:52 +09:00
Inigo Goiri
d973f37054 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 
(cherry picked from commit 2148a8fe64)
 2021-04-02 17:14:34 +08:00
Inigo Goiri
887948d127 HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 
(cherry picked from commit 719b53a79d)
 2021-04-02 17:14:24 +08:00
Surendra Singh Lilhore
02d37c7d22 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 
(cherry picked from commit 743c2e9071)
 2021-04-02 17:14:14 +08:00
Takanobu Asanuma
14fddba715 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 
(cherry picked from commit 736659e0e1)
 2021-04-02 17:13:59 +08:00
Ayush Saxena
2da3356428 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 
(cherry picked from commit fc8a6dd8a9)
 2021-04-02 17:13:00 +08:00
Takanobu Asanuma
f5cc1540b4 HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina. 
(cherry picked from commit ca8e7a7725)
 2021-04-02 17:11:33 +08:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c37)
 2021-04-02 17:11:16 +08:00
Ayush Saxena
5bfc3a4c3a HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu. 
(cherry picked from commit 8a9a674ef1)
 2021-04-02 17:10:56 +08:00
Inigo Goiri
327488d688 HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 
(cherry picked from commit 80b877a72f)
 2021-04-02 14:12:17 +08:00
Inigo Goiri
57d2fff84b HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 
(cherry picked from commit 8b7695bb26)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
1148d3895f HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 
(cherry picked from commit 130f89e068)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
41fad5056c HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina. 
(cherry picked from commit 31b2f687ef)
 2021-04-02 14:06:52 +08:00
Ayush Saxena
5187bd37ae HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 
(cherry picked from commit 37d6582223)
 2021-04-02 14:06:52 +08:00
maobaolong
2b207ea402 HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 
(cherry picked from commit 95c96605b3)
 2021-04-02 14:06:52 +08:00
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6)
 2021-04-02 10:57:05 +09:00
Viraj Jasani
f707d8407a
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:35:44 -07:00
He Xiaoqiao
e5d7b22b8b
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:59:53 +08:00
Brahma Reddy Battula
13878fc06b HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 0665ce9930)
 2021-04-01 09:22:18 +05:30
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
He Xiaoqiao
ab451b8e42 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa1)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
53b024a2b7 HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module 
(cherry picked from commit 9062814bab)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
a407f7de5a HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 
(cherry picked from commit 32895f4f7e)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9d)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb81)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
Harunobu Daikoku
db8af483d1 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b6)
 2021-03-29 11:47:16 +09:00
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf42)
 2021-03-25 13:25:17 +00:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aa)
 2021-03-25 05:09:11 +08:00
Takanobu Asanuma
9a0c5d0ca2 HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina. 
(cherry picked from commit 1fdfaebd98)
 2021-03-24 17:44:29 +08:00
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1)
 2021-03-24 17:44:29 +08:00