big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,055 Commits 2 Branches 2 Tags
Commit Graph

8549 Commits

Author SHA1 Message Date
Michael Kuchenbecker
78723e045a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915ae3e0ced1b4c814a94845f51e655a0c)
 2021-10-21 13:48:38 -07:00
daimin
6019239825 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:02:26 +01:00
Masatake Iwasaki
9e2936f8d1
HADOOP-17424. Replace HTrace with No-Op tracer (#3520) 
(cherry picked from commit 1a205cc3adffa568c814a5241e041b08e2fcd3eb)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tracing/TestTracing.java

Co-authored-by: Siyao Meng <50227127+smengcl@users.noreply.github.com>
 2021-10-12 00:07:09 +09:00
langlaile1221
a35a6ec06b
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c88bcec5acca30aadca36b7d65c4237c)
 2021-10-11 16:22:46 +08:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6b4a654d5e1e7c8e3b4d2ca1a736d53)
 2021-10-07 11:00:19 +09:00
S O'Donnell
f437b3bdaf HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 
(cherry picked from commit a30231ff8e3f1e76009e28d8f8322ffbdeecd061)
 2021-10-06 15:39:34 +01:00
Wei-Chiu Chuang
bb08de559a HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
 2021-09-24 09:13:15 -07:00
Chao Sun
ff26a7700d Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383)" 
This reverts commit 54c40cbf49f2ebf4bbc1976279a6eba7a2c5fe23.
 2021-09-23 15:04:27 -07:00
Ayush Saxena
80d2bda51c
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.(Addendum) 2021-09-23 10:33:16 +05:30
Ayush Saxena
38f529abf7
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-23 08:54:01 +05:30
Ayush Saxena
3355126062
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2021-09-23 08:52:20 +05:30
wangzhaohui
2f73ac1c14 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
 2021-09-15 10:02:13 -07:00
EungsopYoo
51a4a23e37
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
 2021-09-14 16:27:59 +08:00
Symious
c0f32f3cf8 HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 
(cherry picked from commit 7f6553af7546e512f51ba21cbd6cb4e7b73c1961)
 2021-09-13 10:48:09 +08:00
Ahmed Hussein
1f61944e3b HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:27:13 +00:00
Uma Maheswara Rao G
580b6c400b
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit 164608b5465ebc6a2d296376aafbf4b272d9cff0)
 2021-08-31 12:27:43 +08:00
Viraj Jasani
0967483b7c HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 1b9927afe1dba136ea460ec1417514c0d7815504)
 2021-08-25 17:42:56 +09:00
litao
5a1ed37893
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e2bf4c87f12b53e508edf8fee05856a)
 2021-08-19 12:31:52 +08:00
wangzhaohui
7068e18842
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c4d1368e1195132bb32a815cb12934c)
 2021-08-06 18:25:00 +08:00
wangzhaohui
e9ba4f4591 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b733177cf9b8b10088527e34099d73f)
 2021-08-06 09:28:39 +01:00
Siyao Meng
72508e6430 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00b5589fe610ff994d81feba86039297)
 2021-08-03 20:24:18 +01:00
He Xiaoqiao
797c595ba5
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4863499207733a9f4c8c3cd2ec9c9d2)
 2021-07-29 19:04:08 +08:00
jianghuazhu
43f15578fc
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 6f730fd25c9845fea09b98c64b5d59bee5576db2)
 2021-07-29 15:44:02 +08:00
S O'Donnell
d661afc06f HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 
(cherry picked from commit 4eae284827c040cd9fd0eab2e7fd8aaae721326e)
 2021-07-28 15:02:16 +01:00
Akira Ajisaka
9e72b697a2
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit d710ec8d85692e4c2f915db1474b15dba84af666)
 2021-07-26 16:47:26 +09:00
Akira Ajisaka
3f573b6751 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60ceed733bfacb791fc5ed06116720dd0.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488c8da0227b95548c1fe40823b16a44d)
 2021-07-26 01:54:32 +09:00
crossfire
3faae2e06d
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810beb7abb8540c051114488e69a128ea3)
 2021-07-16 11:07:27 +08:00
Jim Brennan
0f2b89b791 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 20:47:02 +00:00
Abhishek Das
450dae7383 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:23:27 -07:00
Artem Smotrakov
c18ad9a007 HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)
 2021-07-10 13:43:31 +09:00
S O'Donnell
705884b87a HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76ca5c206639e2ddea4f3bef393dbb11)
 2021-07-09 10:19:47 +01:00
bshashikant
ab939814a3 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9ab884015290876798a734cbe5c0038b)
 2021-07-09 09:17:10 +01:00
Viraj Jasani
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf521dc9c9660b7dd836fce04d05bab5)
 2021-07-05 13:42:51 +09:00
litao
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e43c7a0b4f16002f5aff9d8c596413a9)
 2021-07-05 08:15:10 +09:00
Viraj Jasani
bf6ed21104 HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:59:25 -07:00
litao
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 
(cherry picked from commit 95454d821c6e4c08d49df195d7c1e0ea1aacafb9)
 2021-06-29 18:15:56 +08:00
Viraj Jasani
8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073d76bef5c6ad3819e7710317733bd7a)
 2021-06-29 18:13:00 +08:00
litao
4ca49fcd44 HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 16:00:47 +05:30
litao
5fc61d2e56 HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 
(cherry picked from commit fdef2b4ccacb8753aac0f5625505181c9b4dc154)
 2021-06-24 13:27:33 +09:00
Viraj Jasani
68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 
(cherry picked from commit d9fbb3c5082baf301b082f51eea2f8a2e25e8715)
 2021-06-23 11:24:00 +08:00
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79cc1ad2596cbf509a0cde14acc5ad6b)
 2021-06-21 10:28:05 +09:00
litao
7d202e8365 HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:24:03 +05:30
Ahmed Hussein
92ade1f6f9 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:49 +05:30
litao
8d241482bd HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:38 +05:30
Hui Fei
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3113) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 14:41:02 +08:00
Ahmed Hussein
840612b3b1 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 
(cherry picked from commit 747883af4d6cfb41b1f6bd9a4f9c9d246d021048)
 2021-06-14 14:37:27 +00:00
litao
52d055a7eb HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0e82d8bcd34867aea915c7838541fb7)
 2021-06-11 17:33:06 +09:00
litao
02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17ace8b45ee0f3c3967d87893765eea61)
 2021-06-08 16:59:59 +09:00
litao
46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 (#3021) 2021-06-08 00:10:37 -07:00
huhaiyang
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e52839911ab4e0b97fd269c53bf7d1380f)
 2021-06-01 15:30:50 +08:00