big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,186 Commits 2 Branches 2 Tags 264 MiB
164a4e63e0
Commit Graph

5880 Commits

Author SHA1 Message Date
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc)
 2021-02-03 08:58:10 -06:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:19:21 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:09:17 +09:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0)
 2021-01-04 11:28:26 -08:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 17:50:47 +00:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
S O'Donnell
1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c)
 2020-12-11 21:05:00 +00:00
Ahmed Hussein
16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
 2020-12-10 13:50:32 -08:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Hui Fei
cb2dce30d4 HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 10:40:14 +08:00
crossfire
edd9b659ca HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
 2020-12-07 15:50:38 -08:00
Jim Brennan
6a5864ee4a HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 2020-12-07 21:19:36 +00:00
Ahmed Hussein
2249edb001 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69)
 2020-12-02 21:40:17 +00:00
Ahmed Hussein
0f18a19d97 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7)
 2020-12-02 20:12:31 +00:00
Ahmed Hussein
a660714387 HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82f)
 2020-12-01 21:16:49 +00:00
Takanobu Asanuma
d78bf82722 HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d)
 2020-11-13 11:05:20 +09:00
Yizhong Zhang
12bfd9194a HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794)
 2020-11-10 09:14:12 -08:00
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 14:52:05 -08:00
Hui Fei
b8cc09d7c7 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:25:03 +08:00
Wei-Chiu Chuang
c8eb86ee3d HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
 2020-10-23 12:06:23 -07:00
Kihwal Lee
02709cb054 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 19:41:38 -05:00
Kihwal Lee
cf932a7e2d HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-22 09:55:28 -05:00
Kihwal Lee
433434d509 HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:16:47 -05:00
S O'Donnell
5f321df0a0 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-22 12:00:02 +01:00
hemanthboyina
94e5c5257f HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 09:40:36 +05:30
Mingliang Liu
4eccdd950f
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell 2020-09-12 01:50:28 -07:00
Stephen O'Donnell
a9ce6001ea HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2ffe00fc46)
 2020-08-26 23:07:27 -07:00
Ayush Saxena
9eaa3520e6 HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu. 
(cherry picked from commit 0fe49036e5)
 2020-08-26 23:07:22 -07:00
Uma Maheswara Rao G
ba0eca6a2c HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dd013f2fdf)
 2020-08-25 12:00:52 -07:00
S O'Donnell
97d6058aa6 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 15:37:52 +01:00
hemanthboyina
60d167d080 HDFS-14504. Rename with Snapshots does not honor quota limit. 
(cherry picked from commit 83fd15b412)
 2020-08-23 16:12:36 +05:30
Chen Liang
b1292bdaef HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d)
 2020-08-20 15:27:52 -07:00
S O'Donnell
42154f04e3 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 
(cherry picked from commit 10716040a8)
 2020-08-12 09:33:14 +01:00
hemanthboyina
b5d712251c HDFS-15503. File and directory permissions are not able to be modified from WebUI. 
(cherry picked from commit 82f3ffcd64)
 2020-08-03 23:43:29 +05:30
Uma Maheswara Rao G
aea1a8e2bd HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:30:30 -07:00
Chen Liang
c8c40be761 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 15:22:34 -07:00
Ahmed Hussein
9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Ahmed Hussein
5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 09:57:36 -05:00
Shashikant Banerjee
292e22578a HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 
(cherry picked from commit 82343790ee)
 2020-07-15 13:13:27 +01:00
He Xiaoqiao
e7e7a6d503 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:31:50 +08:00
Shanyu Zhao
10c9df1d0a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 11:44:42 +08:00
Ayush Saxena
b8d9218189 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:25:29 +05:30
Akira Ajisaka
3e9422d1c7
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:04:47 +09:00
Stephen O'Donnell
7613191fcd HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 630ac9e7e5fdbc3ce86476cf0167255ab9b0470a)
 2020-06-18 06:45:28 -07:00