big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,276 Commits 2 Branches 2 Tags 264 MiB
b26d75ce37
Commit Graph

8940 Commits

Author SHA1 Message Date
Ayush Saxena
b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f66)
 2021-03-23 15:34:48 +00:00
Ayush Saxena
968c95bfba HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit ae13a5ccbe)
 2021-03-23 15:34:36 +00:00
Ayush Saxena
d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena
793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58)
 2021-03-22 12:35:11 +00:00
litao
bc2bdee76c HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 
(cherry picked from commit 343ce8a8c9)
 2021-03-22 20:30:53 +09:00
Ayush Saxena
0b43c65816 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad)
 2021-03-19 18:50:31 +00:00
Ayush Saxena
9665a0be73 HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:18 +05:30
Viraj Jasani
17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma
aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:15:53 +09:00
touchida
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 12:01:15 +09:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:07:48 +08:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:00:25 -08:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:35:45 +09:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)
 2021-03-01 11:50:23 +00:00
Surendra Singh Lilhore
a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
 2021-02-26 10:49:16 +08:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:04:52 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
 2021-02-22 18:14:18 +08:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
 2021-02-22 15:24:37 +08:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a9)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Akira Ajisaka
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b1733)
 2021-02-16 14:52:31 +09:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:32:14 +00:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
Jim Brennan
62389a5a04 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 22:16:36 +00:00
fermi
ae80fc2477 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:05:18 +08:00
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc)
 2021-02-03 08:58:10 -06:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
f97709beaa
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:52:08 +08:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:19:21 +00:00
Jim Brennan
2c70705575 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 16:51:40 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:09:17 +09:00
He Xiaoqiao
26cd02fb29
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-19 23:29:45 +08:00
Masatake Iwasaki
89452682a5
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:20:49 +09:00
touchida
75f34a156f
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:25:10 +09:00
He Xiaoqiao
e95ee67632
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:10 +08:00
Ahmed Hussein
18e2835766 HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 
(cherry picked from commit 77435a025e)
 2021-01-08 19:29:14 +00:00
Wei-Chiu Chuang
94c126cc9e HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
 2021-01-04 20:56:18 -08:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0)
 2021-01-04 11:28:26 -08:00
Liang-Chi Hsieh
060594259c HDFS-15690. Add lz4-java as test dependency (#2481) 2020-12-29 13:36:04 -08:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 17:50:47 +00:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
S O'Donnell
1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c)
 2020-12-11 21:05:00 +00:00
Ahmed Hussein
16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
 2020-12-10 13:50:32 -08:00