big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,868 Commits 2 Branches 2 Tags 264 MiB
e5516cdfaf
Commit Graph

8499 Commits

Author SHA1 Message Date
Ashutosh Gupta
bad3a0964c HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:25:56 +09:00
jianghuazhu
b623acff5a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:25:00 +09:00
jianghuazhu
a4496a0187 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 11:32:01 +00:00
litao
e60b8a0358 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:51:40 +09:00
litao
75af1d67e4 HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2021-12-16 13:31:17 +09:00
Ayush Saxena
37ced452d9 HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:23:24 +09:00
litao
8559ba46c2 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2021-12-14 11:53:46 +09:00
GuoPhilipse
2720479988 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2021-12-10 16:33:33 +09:00
Akira Ajisaka
35c5c6bb83 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2021-12-10 01:38:26 +09:00
qinyuren
0dfb4eb602
HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:33:03 +09:00
Takanobu Asanuma
a67f4dc71a Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2021-12-09 16:36:54 +09:00
qinyuren
55c0e676fd HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2021-12-09 13:26:00 +09:00
LeonGao
63495978b2 HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2021-12-06 16:18:25 +09:00
Takanobu Asanuma
f6b3361b4b HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2021-12-06 11:05:54 +09:00
huhaiyang
a24c98163f
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2021-12-04 02:27:35 +09:00
huhaiyang
bcbe4fbfa7
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2021-12-04 02:24:40 +09:00
bitterfox
2cdecd420a
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:35:46 +09:00
litao
f5e15dfb11 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2021-11-29 17:54:58 +09:00
litao
b428d534fb HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2021-11-26 19:53:34 +09:00
Akira Ajisaka
4a032bc88d Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2021-11-25 14:24:30 +09:00
Viraj Jasani
6094e1ec9a
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2021-11-25 14:15:10 +09:00
Viraj Jasani
a9125bba24 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:28:40 +08:00
Ayush Saxena
3b79e0f794
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2021-11-25 01:54:14 +05:30
Viraj Jasani
450efaeabe
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:23 +09:00
Mukul Kumar Singh
97de402970 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:39:16 +00:00
litao
d746e2cdd8
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2021-11-23 15:43:02 +08:00
litao
90085eb4cb HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2021-11-22 17:14:17 +09:00
GuoPhilipse
6b78a753e3
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2021-11-22 07:45:42 +09:00
litao
166c38bcbf HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:06 +09:00
GuoPhilipse
8166bcc668
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:31 +09:00
Viraj Jasani
2f587bf15a
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:13 +09:00
litao
4036213273
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:25 +09:00
Viraj Jasani
4e77f772fc HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2021-11-17 22:19:38 +09:00
litao
026d5860cb
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2021-11-17 11:06:53 +09:00
Chao Sun
e079fa6577 Preparing for 3.3.3 development 2021-11-16 16:02:34 -08:00
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang
5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao
c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao
808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
jianghuazhu
46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00
litao
26b7bc75ed HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:09:05 +09:00
jianghuazhu
4d04efb900 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:50:31 +09:00
daimin
2844b98558 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)
 2021-11-03 20:41:55 +00:00
Stephen O'Donnell
9cf841b1a6 HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 
(cherry picked from commit 2f35cc36cd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
 2021-11-02 11:17:59 +00:00
jianghuazhu
5fbd9d333e HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 618fea27d2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
 2021-11-02 01:04:33 +09:00
litao
7e57fc053a HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:28:40 +09:00
daimin
847cff7d91
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:16:06 +05:30
GuoPhilipse
c4f1db1966
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:02:21 +05:30
Michael Kuchenbecker
78723e045a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:48:38 -07:00
daimin
6019239825 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:02:26 +01:00
Masatake Iwasaki
9e2936f8d1
HADOOP-17424. Replace HTrace with No-Op tracer (#3520) 
(cherry picked from commit 1a205cc3ad)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
 2021-10-12 00:07:09 +09:00
langlaile1221
a35a6ec06b
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
 2021-10-11 16:22:46 +08:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6)
 2021-10-07 11:00:19 +09:00
S O'Donnell
f437b3bdaf HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 
(cherry picked from commit a30231ff8e)
 2021-10-06 15:39:34 +01:00
Wei-Chiu Chuang
bb08de559a HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
 2021-09-24 09:13:15 -07:00
Chao Sun
ff26a7700d Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383)" 
This reverts commit 54c40cbf49.
 2021-09-23 15:04:27 -07:00
Ayush Saxena
80d2bda51c
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.(Addendum) 2021-09-23 10:33:16 +05:30
Ayush Saxena
38f529abf7
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-23 08:54:01 +05:30
Ayush Saxena
3355126062
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2021-09-23 08:52:20 +05:30
wangzhaohui
2f73ac1c14 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
 2021-09-15 10:02:13 -07:00
EungsopYoo
51a4a23e37
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8b)
 2021-09-14 16:27:59 +08:00
Symious
c0f32f3cf8 HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 
(cherry picked from commit 7f6553af75)
 2021-09-13 10:48:09 +08:00
Ahmed Hussein
1f61944e3b HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:27:13 +00:00
Uma Maheswara Rao G
580b6c400b
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit 164608b546)
 2021-08-31 12:27:43 +08:00
Viraj Jasani
0967483b7c HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 1b9927afe1)
 2021-08-25 17:42:56 +09:00
litao
5a1ed37893
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e)
 2021-08-19 12:31:52 +08:00
wangzhaohui
7068e18842
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c)
 2021-08-06 18:25:00 +08:00
wangzhaohui
e9ba4f4591 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b)
 2021-08-06 09:28:39 +01:00
Siyao Meng
72508e6430 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00)
 2021-08-03 20:24:18 +01:00
He Xiaoqiao
797c595ba5
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4)
 2021-07-29 19:04:08 +08:00
jianghuazhu
43f15578fc
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 6f730fd25c)
 2021-07-29 15:44:02 +08:00
S O'Donnell
d661afc06f HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 
(cherry picked from commit 4eae284827)
 2021-07-28 15:02:16 +01:00
Akira Ajisaka
9e72b697a2
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit d710ec8d85)
 2021-07-26 16:47:26 +09:00
Akira Ajisaka
3f573b6751 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488)
 2021-07-26 01:54:32 +09:00
crossfire
3faae2e06d
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810b)
 2021-07-16 11:07:27 +08:00
Jim Brennan
0f2b89b791 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 20:47:02 +00:00
Abhishek Das
450dae7383 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b5)
 2021-07-13 18:23:27 -07:00
Artem Smotrakov
c18ad9a007 HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)
 2021-07-10 13:43:31 +09:00
S O'Donnell
705884b87a HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76)
 2021-07-09 10:19:47 +01:00
bshashikant
ab939814a3 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9a)
 2021-07-09 09:17:10 +01:00
Viraj Jasani
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf)
 2021-07-05 13:42:51 +09:00
litao
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e4)
 2021-07-05 08:15:10 +09:00
Viraj Jasani
bf6ed21104 HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:59:25 -07:00
litao
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 
(cherry picked from commit 95454d821c)
 2021-06-29 18:15:56 +08:00
Viraj Jasani
8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073)
 2021-06-29 18:13:00 +08:00
litao
4ca49fcd44 HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 16:00:47 +05:30
litao
5fc61d2e56 HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 
(cherry picked from commit fdef2b4cca)
 2021-06-24 13:27:33 +09:00
Viraj Jasani
68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 
(cherry picked from commit d9fbb3c508)
 2021-06-23 11:24:00 +08:00
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
 2021-06-21 10:28:05 +09:00
litao
7d202e8365 HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:24:03 +05:30
Ahmed Hussein
92ade1f6f9 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:49 +05:30
litao
8d241482bd HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:38 +05:30
Hui Fei
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3113) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 14:41:02 +08:00
Ahmed Hussein
840612b3b1 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 
(cherry picked from commit 747883af4d)
 2021-06-14 14:37:27 +00:00
litao
52d055a7eb HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0)
 2021-06-11 17:33:06 +09:00
litao
02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a)
 2021-06-08 16:59:59 +09:00
litao
46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 (#3021) 2021-06-08 00:10:37 -07:00
huhaiyang
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e528)
 2021-06-01 15:30:50 +08:00