big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,849 Commits 2 Branches 2 Tags
Commit Graph

8486 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e180902956fd547ad6a7b440588fdf583870c2)
 2021-05-02 11:12:37 +09:00
Hui Fei
9aa6106689 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:10:04 -07:00
litao
0e9042fed5 HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit b968fa0957ddd0f59c9adfd068507722db7207c6)
 2021-04-28 18:52:05 +09:00
Hui Fei
842e05ba53 HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495) 
(cherry picked from commit 68442b48c8a044b1ead89e22003b6db4d300bfad)
 2021-04-28 18:51:50 +09:00
Ayush Saxena
f809b863fe HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 
(cherry picked from commit 6e416a83d1e674ecd018d1db74a2d88e738deb40)
 2021-04-28 18:51:43 +09:00
Ayush Saxena
187b4a63a4 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6c6ce9215ea998a2f544b2eabca4340)
 2021-04-27 23:18:12 +09:00
litao
f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 (#2940) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-27 22:39:25 +09:00
Stephen O'Donnell
f6efb58b07 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c291a6250b077da32a49dbb35f3b78bf7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-04-26 11:23:56 +01:00
Takanobu Asanuma
a5f038b3f6 HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 
(cherry picked from commit 01bad0e92a1f402930bb4c0d18dd605bdf5627e5)
 2021-04-26 13:31:51 +09:00
Brahma Reddy Battula
99dfd3b2d0 HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2621d3f15bfae9b3820f70c0dfda015b1d474a97)
 2021-04-26 11:40:16 +08:00
Viraj Jasani
96a1dfa313
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 12:57:23 +09:00
zhangshuyan0
9f2db2c9fd
HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-21 18:32:56 +08:00
litao
29414871bd
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2928) 2021-04-21 10:41:13 +09:00
hemanthboyina
90f63cb12c HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 
(cherry picked from commit 16a20503cacf12c3d8e27ba90820384f58bed06f)
 2021-04-20 16:21:57 +01:00
Ayush Saxena
c93a2f6789 HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 
(cherry picked from commit c84e6beada4e604175f7f138c9878a29665a8c47)
 2021-04-20 12:13:39 +01:00
litao
dc830cf277 HDFS-15970. Print network topology on the web (#2896) 
(cherry picked from commit 85a3532849d81ca929bc52cb7ca905c4f53652ec)
 2021-04-19 21:22:47 +09:00
Hui Fei
4c0b833bab HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 
(cherry picked from commit 0d4210411946d7f214d90e047b9f7bb9b7b82dfb)
 2021-04-16 14:07:15 +09:00
Wei-Chiu Chuang
bfba6f1f3c
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e414abf8c94f7bd4719053efa3b538a)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-12 23:55:32 -07:00
Stephen O'Donnell
0e6cd352b5
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. (#2813) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2021-04-09 10:25:14 +01:00
Viraj Jasani
4994b73eeb HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 
(cherry picked from commit 56bd968fb434c7d2ae7ac9e7659d0ca682340b2a)
 2021-04-09 11:13:39 +09:00
He Xiaoqiao
df99ac0399 HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df4199791a9093324e089b4a5cc880be8d6ff)
 2021-04-08 13:15:37 +01:00
Stephen O'Donnell
ef95f7a963 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:15:31 +01:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
 2021-04-08 17:20:08 +08:00
Viraj Jasani
8b4b3d6fe6 HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b540be3ce15642ab8be463df87a4e4e)
 2021-04-06 14:18:31 +09:00
Viraj Jasani
2b5fd341b9 HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:20:56 +05:30
Inigo Goiri
d973f37054 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 
(cherry picked from commit 2148a8fe645333444c4e8110bb56acf0fb8e41b4)
 2021-04-02 17:14:34 +08:00
Inigo Goiri
887948d127 HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 
(cherry picked from commit 719b53a79dc169a8c52229831dcb011935a8a151)
 2021-04-02 17:14:24 +08:00
Surendra Singh Lilhore
02d37c7d22 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 
(cherry picked from commit 743c2e9071f4a73e0196ad4ca005b767758642b9)
 2021-04-02 17:14:14 +08:00
Inigo Goiri
57d2fff84b HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 
(cherry picked from commit 8b7695bb2628574b4450bac19c12b29db9ee0628)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
1148d3895f HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 
(cherry picked from commit 130f89e068ca346a44fa6619ae0726c1e5cc5d06)
 2021-04-02 14:06:52 +08:00
Ayush Saxena
5187bd37ae HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 
(cherry picked from commit 37d65822235fe8285d10232589aba39ededd3be1)
 2021-04-02 14:06:52 +08:00
maobaolong
2b207ea402 HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 
(cherry picked from commit 95c96605b30cc31839a04bd5d4061a2c89e4c09c)
 2021-04-02 14:06:52 +08:00
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 10:57:05 +09:00
Viraj Jasani
f707d8407a
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:35:44 -07:00
He Xiaoqiao
e5d7b22b8b
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:59:53 +08:00
Brahma Reddy Battula
13878fc06b HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 0665ce99308aba1277d8f36bad9308062ad4b6ea)
 2021-04-01 09:22:18 +05:30
He Xiaoqiao
ab451b8e42 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9dd12b874ced1798f0c3ad0c160cb465)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb8139a08b92b5a94adfde544de978137c)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b071fe5b943542827fd8a496b137fa9)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf429d28561dd9c582f5c55451213d89a4)
 2021-03-25 13:25:17 +00:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aad1b5a131d366aa5395f2904f0f6ecd)
 2021-03-25 05:09:11 +08:00
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1bf4f0a45571c9fc4361e4e8a05f7ee)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
580634882e HDFS-14546. Document block placement policies. Contributed by Amithsha. 
(cherry picked from commit 17ffcab5f621400bd8bb47dc8fb29365f6e24ebd)
 2021-03-24 17:44:29 +08:00
Karthik Palanisamy
c96821ece8 HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 
(cherry picked from commit 735e85a1d2036585075b81d1fb2fd302e89eb363)
 2021-03-24 17:44:29 +08:00
sunlisheng
a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944e628345109b43cba2cd4d49ca8cc6b)
 2021-03-24 14:04:47 +08:00
Ayush Saxena
94766fdb13 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 
(cherry picked from commit ff84a57483320abf81bf941b81d13d5bee14084b)
 2021-03-24 14:04:47 +08:00