big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,704 Commits 2 Branches 2 Tags 264 MiB
97de402970
Commit Graph

9125 Commits

Author SHA1 Message Date
Mukul Kumar Singh
97de402970 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:39:16 +00:00
litao
d746e2cdd8
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2021-11-23 15:43:02 +08:00
litao
90085eb4cb HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2021-11-22 17:14:17 +09:00
GuoPhilipse
6b78a753e3
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2021-11-22 07:45:42 +09:00
litao
166c38bcbf HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:06 +09:00
GuoPhilipse
8166bcc668
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:31 +09:00
Viraj Jasani
2f587bf15a
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:13 +09:00
litao
4036213273
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:25 +09:00
Viraj Jasani
4e77f772fc HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2021-11-17 22:19:38 +09:00
litao
026d5860cb
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2021-11-17 11:06:53 +09:00
Chao Sun
e079fa6577 Preparing for 3.3.3 development 2021-11-16 16:02:34 -08:00
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang
5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao
c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao
808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
litao
0db04fab20 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:18 +09:00
jianghuazhu
46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00
litao
26b7bc75ed HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:09:05 +09:00
jianghuazhu
4d04efb900 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:50:31 +09:00
daimin
2844b98558 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)
 2021-11-03 20:41:55 +00:00
Stephen O'Donnell
9cf841b1a6 HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 
(cherry picked from commit 2f35cc36cd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
 2021-11-02 11:17:59 +00:00
jianghuazhu
5fbd9d333e HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 618fea27d2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java
 2021-11-02 01:04:33 +09:00
litao
7e57fc053a HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:28:40 +09:00
daimin
847cff7d91
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:16:06 +05:30
GuoPhilipse
c4f1db1966
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:02:21 +05:30
Michael Kuchenbecker
78723e045a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:48:38 -07:00
daimin
6019239825 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:02:26 +01:00
Ayush Saxena
5b1cf17f88
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:53 +05:30
Masatake Iwasaki
9e2936f8d1
HADOOP-17424. Replace HTrace with No-Op tracer (#3520) 
(cherry picked from commit 1a205cc3ad)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
 2021-10-12 00:07:09 +09:00
langlaile1221
a35a6ec06b
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
 2021-10-11 16:22:46 +08:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6)
 2021-10-07 11:00:19 +09:00
S O'Donnell
f437b3bdaf HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 
(cherry picked from commit a30231ff8e)
 2021-10-06 15:39:34 +01:00
AngersZhuuuu
4475d8bfe7
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4)
 2021-09-29 18:36:36 +08:00
He Xiaoqiao
06954af6f0
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe)
 2021-09-29 18:36:36 +08:00
Wei-Chiu Chuang
bb08de559a HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
 2021-09-24 09:13:15 -07:00
Akira Ajisaka
f1c0dc84cd
HDFS-15977. Call explicit_bzero only if it is available. (#2914) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit f0241ec216)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/x-platform/syscall_linux.cc
 2021-09-24 11:50:12 +09:00
Chao Sun
ff26a7700d Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383)" 
This reverts commit 54c40cbf49.
 2021-09-23 15:04:27 -07:00
Ayush Saxena
80d2bda51c
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.(Addendum) 2021-09-23 10:33:16 +05:30
Ayush Saxena
38f529abf7
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-23 08:54:01 +05:30
Ayush Saxena
3355126062
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2021-09-23 08:52:20 +05:30
wangzhaohui
2f73ac1c14 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
 2021-09-15 10:02:13 -07:00
EungsopYoo
51a4a23e37
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8b)
 2021-09-14 16:27:59 +08:00
Symious
c0f32f3cf8 HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 
(cherry picked from commit 7f6553af75)
 2021-09-13 10:48:09 +08:00
Symious
8affaa6312 HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 
(cherry picked from commit c0890e6d04)
 2021-09-10 10:01:27 +08:00
Ahmed Hussein
1f61944e3b HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:27:13 +00:00
Uma Maheswara Rao G
580b6c400b
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit 164608b546)
 2021-08-31 12:27:43 +08:00
Viraj Jasani
0967483b7c HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 1b9927afe1)
 2021-08-25 17:42:56 +09:00
litao
5a1ed37893
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e)
 2021-08-19 12:31:52 +08:00