big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,729 Commits 2 Branches 2 Tags 264 MiB
0dfb4eb602
Commit Graph

8440 Commits

Author SHA1 Message Date
qinyuren
0dfb4eb602
HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:33:03 +09:00
Takanobu Asanuma
a67f4dc71a Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2021-12-09 16:36:54 +09:00
qinyuren
55c0e676fd HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2021-12-09 13:26:00 +09:00
LeonGao
63495978b2 HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2021-12-06 16:18:25 +09:00
Takanobu Asanuma
f6b3361b4b HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2021-12-06 11:05:54 +09:00
huhaiyang
a24c98163f
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2021-12-04 02:27:35 +09:00
huhaiyang
bcbe4fbfa7
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2021-12-04 02:24:40 +09:00
bitterfox
2cdecd420a
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:35:46 +09:00
litao
f5e15dfb11 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2021-11-29 17:54:58 +09:00
litao
b428d534fb HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2021-11-26 19:53:34 +09:00
Akira Ajisaka
4a032bc88d Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2021-11-25 14:24:30 +09:00
Viraj Jasani
6094e1ec9a
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2021-11-25 14:15:10 +09:00
Viraj Jasani
a9125bba24 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:28:40 +08:00
Ayush Saxena
3b79e0f794
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2021-11-25 01:54:14 +05:30
Viraj Jasani
450efaeabe
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:23 +09:00
Mukul Kumar Singh
97de402970 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:39:16 +00:00
litao
d746e2cdd8
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2021-11-23 15:43:02 +08:00
litao
90085eb4cb HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2021-11-22 17:14:17 +09:00
GuoPhilipse
6b78a753e3
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2021-11-22 07:45:42 +09:00
litao
166c38bcbf HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:06 +09:00
GuoPhilipse
8166bcc668
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:31 +09:00
Viraj Jasani
2f587bf15a
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:13 +09:00
litao
4036213273
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:25 +09:00
Viraj Jasani
4e77f772fc HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2021-11-17 22:19:38 +09:00
litao
026d5860cb
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2021-11-17 11:06:53 +09:00
Chao Sun
e079fa6577 Preparing for 3.3.3 development 2021-11-16 16:02:34 -08:00
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang
5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao
c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao
808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
jianghuazhu
46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00
litao
26b7bc75ed HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:09:05 +09:00
jianghuazhu
4d04efb900 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:50:31 +09:00
daimin
2844b98558 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)
 2021-11-03 20:41:55 +00:00
Stephen O'Donnell
9cf841b1a6 HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 
(cherry picked from commit 2f35cc36cd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
 2021-11-02 11:17:59 +00:00
jianghuazhu
5fbd9d333e HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 618fea27d2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
 2021-11-02 01:04:33 +09:00
litao
7e57fc053a HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:28:40 +09:00
daimin
847cff7d91
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:16:06 +05:30
GuoPhilipse
c4f1db1966
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:02:21 +05:30
Michael Kuchenbecker
78723e045a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:48:38 -07:00
daimin
6019239825 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:02:26 +01:00
Masatake Iwasaki
9e2936f8d1
HADOOP-17424. Replace HTrace with No-Op tracer (#3520) 
(cherry picked from commit 1a205cc3ad)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
 2021-10-12 00:07:09 +09:00
langlaile1221
a35a6ec06b
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
 2021-10-11 16:22:46 +08:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6)
 2021-10-07 11:00:19 +09:00
S O'Donnell
f437b3bdaf HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 
(cherry picked from commit a30231ff8e)
 2021-10-06 15:39:34 +01:00
Wei-Chiu Chuang
bb08de559a HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
 2021-09-24 09:13:15 -07:00
Chao Sun
ff26a7700d Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383)" 
This reverts commit 54c40cbf49.
 2021-09-23 15:04:27 -07:00
Ayush Saxena
80d2bda51c
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.(Addendum) 2021-09-23 10:33:16 +05:30