Jim Brennan
2c70705575
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
...
(cherry picked from commit 6184ab07f6
)
2021-01-28 16:51:40 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
...
(cherry picked from commit 45b3a84709
)
2021-01-21 10:09:17 +09:00
He Xiaoqiao
26cd02fb29
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-19 23:29:45 +08:00
Masatake Iwasaki
89452682a5
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c
)
2021-01-14 16:20:49 +09:00
touchida
75f34a156f
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba
)
2021-01-13 11:25:10 +09:00
He Xiaoqiao
e95ee67632
Make upstream aware of 3.2.2 release.
2021-01-09 18:07:10 +08:00
Ahmed Hussein
18e2835766
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
...
(cherry picked from commit 77435a025e
)
2021-01-08 19:29:14 +00:00
Wei-Chiu Chuang
6340ac857b
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
...
(cherry picked from commit 66ee0a6df0
)
2021-01-04 11:28:26 -08:00
Liang-Chi Hsieh
060594259c
HDFS-15690. Add lz4-java as test dependency ( #2481 )
2020-12-29 13:36:04 -08:00
Ahmed Hussein
be508718d8
HDFS-15704. Mitigate lease monitor's rapid infinite loop. ( #2511 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit c2672bb234
)
2020-12-17 17:50:47 +00:00
Ayush Saxena
faf26f70aa
HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
2020-12-14 20:41:06 -08:00
S O'Donnell
1a63df86e2
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
...
(cherry picked from commit 9ed737001c
)
2020-12-11 21:05:00 +00:00
Ahmed Hussein
16ca8b7d20
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
...
(cherry picked from commit be35fa186c
)
2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 ) ( #2538 )
...
(cherry picked from commit 1b29c9bfee
)
Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
...
(cherry picked from commit 9bd3c9bc50
)
2020-12-10 13:50:32 -08:00
Ayush Saxena
8378ab9f92
HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505
2020-12-10 05:50:55 +05:30
Ahmed Hussein
e02b179c4c
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
...
(cherry picked from commit 01383a2172
)
2020-12-08 19:28:44 +00:00
Hui Fei
cb2dce30d4
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-08 10:40:14 +08:00
crossfire
edd9b659ca
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
...
(cherry picked from commit 40f7543a6d
)
2020-12-07 15:50:38 -08:00
Jim Brennan
6a5864ee4a
HDFS-15707. NNTop counts don't add up as expected. ( #2516 ) Contributed by Ahmed Hussein and Daryn Sharp
2020-12-07 21:19:36 +00:00
Ahmed Hussein
2249edb001
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit 60201cbf69
)
2020-12-02 21:40:17 +00:00
Ahmed Hussein
0f18a19d97
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit 2b5b556dd7
)
2020-12-02 20:12:31 +00:00
Ahmed Hussein
a660714387
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
...
(cherry picked from commit 918ba9e82f
)
2020-12-01 21:16:49 +00:00
Takanobu Asanuma
d78bf82722
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
...
(cherry picked from commit f56cd88d7d
)
2020-11-13 11:05:20 +09:00
Yizhong Zhang
12bfd9194a
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
...
(cherry picked from commit 82de07c794
)
2020-11-10 09:14:12 -08:00
Ayush Saxena
cb11fd8793
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit d07dc7afb4
)
2020-11-03 14:52:05 -08:00
Ahmed Hussein
d4129c6794
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4
)
2020-10-27 20:49:15 +09:00
Ahmed Hussein
1429c65533
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails ( #2404 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d332
)
2020-10-27 16:58:04 +09:00
Hui Fei
b8cc09d7c7
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
...
(cherry picked from commit 12c908c827
)
2020-10-26 22:25:03 +08:00
Wei-Chiu Chuang
c8eb86ee3d
HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.
...
(cherry picked from commit 74634eb002
)
2020-10-23 12:06:23 -07:00
Inigo Goiri
c55d639e8f
HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.
...
(cherry picked from commit e7aa4da0a2
)
2020-10-23 09:48:23 -07:00
Kihwal Lee
02709cb054
HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.
...
(cherry picked from commit da1b6e3cc2
)
2020-10-22 19:41:38 -05:00
Kihwal Lee
cf932a7e2d
HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.
2020-10-22 09:55:28 -05:00
Ayush Saxena
54c40cbf49
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same ( #2383 )
...
Contributed by Gabor Bota.
2020-10-17 01:34:01 +05:30
Kihwal Lee
433434d509
HDFS-15627. Log delete audits before collecting blocks.
...
(cherry picked from commit 740a2c4635
)
2020-10-16 11:16:47 -05:00
Mukund Thakur
351b1498d3
HDFS-15626. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
Change-Id: I6cec41c20467920cf21f169653553535414b2680
2020-10-13 13:32:35 +01:00
Konstantin V Shvachko
b6423d2780
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit b3786d6c3c
)
2020-10-12 17:38:42 -07:00
Hui Fei
ed19f63998
HADOOP-17277. Correct spelling errors for separator ( #2322 )
...
Contributed by Hui Fei.
(cherry picked from commit 474fa80bfb
)
2020-09-23 15:39:51 +09:00
S O'Donnell
5f321df0a0
HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell
2020-09-22 12:00:02 +01:00
Uma Maheswara Rao G
2ce5846bfa
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. ( #2312 ). Contributed by Uma Maheswara Rao G.
2020-09-18 15:23:35 +05:30
Uma Maheswara Rao G
1fc1b34633
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured ( #2275 ) Contributed by Uma Maheswara Rao G.
2020-09-18 15:23:12 +05:30
Uma Maheswara Rao G
2d9c5395ef
HDFS-15578: Fix the rename issues with fallback fs enabled ( #2305 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit e4cb0d3514
)
2020-09-16 23:01:03 -07:00
hemanthboyina
94e5c5257f
HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.
2020-09-17 09:40:36 +05:30
Uma Maheswara Rao G
1195dac55e
HDFS-15529: getChildFilesystems should include fallback fs as well ( #2234 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit b3660d0147
)
2020-09-12 20:48:59 -07:00
Uma Maheswara Rao G
bfa145dd7c
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. ( #2298 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit d2779de3f5
)
2020-09-12 20:43:34 -07:00
zz
2d5ca83078
HADOOP-15891. provide Regex Based Mount Point In Inode Tree ( #2185 ). Contributed by Zhenzhao Wang.
...
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
(cherry picked from commit 12a316cdf9
)
2020-09-12 20:42:06 -07:00
Mingliang Liu
4eccdd950f
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell
2020-09-12 01:50:28 -07:00
hemanthboyina
a2bc0dcd21
HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.
...
(cherry picked from commit 2a6fdcedeb
)
2020-09-11 18:22:49 +05:30
Steve Loughran
1b9109d237
HDFS-15471. TestHDFSContractMultipartUploader failing ( #2252 )
...
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
Change-Id: If6a82706f3ea6d802bc6da03c2a2ca734e30388f
2020-08-28 15:47:06 +01:00
Stephen O'Donnell
a9ce6001ea
HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2ffe00fc46
)
2020-08-26 23:07:27 -07:00
Ayush Saxena
9eaa3520e6
HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.
...
(cherry picked from commit 0fe49036e5
)
2020-08-26 23:07:22 -07:00
Uma Maheswara Rao G
ba0eca6a2c
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. ( #2229 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit dd013f2fdf
)
2020-08-25 12:00:52 -07:00
S O'Donnell
97d6058aa6
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
...
(cherry picked from commit 6a49bf9bff
)
2020-08-25 15:37:52 +01:00
hemanthboyina
60d167d080
HDFS-14504. Rename with Snapshots does not honor quota limit.
...
(cherry picked from commit 83fd15b412
)
2020-08-23 16:12:36 +05:30
Chen Liang
b1292bdaef
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
...
(cherry picked from commit f734455e5d
)
2020-08-20 15:27:52 -07:00
Uma Maheswara Rao G
99b120a06e
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false ( #2205 )
...
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false
* Used LambdaTestUtils#intercept in test
2020-08-13 14:32:19 +01:00
S O'Donnell
42154f04e3
HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang
...
(cherry picked from commit 10716040a8
)
2020-08-12 09:33:14 +01:00
Ayush Saxena
2943e6650f
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:20:42 +05:30
hemanthboyina
b5d712251c
HDFS-15503. File and directory permissions are not able to be modified from WebUI.
...
(cherry picked from commit 82f3ffcd64
)
2020-08-03 23:43:29 +05:30
Uma Maheswara Rao G
4fe491d10e
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. ( #2160 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ac9a07b51a
)
2020-07-31 01:31:37 -07:00
Uma Maheswara Rao G
ae8261c671
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links ( #2132 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 3e70006639
)
2020-07-31 01:31:15 -07:00
Uma Maheswara Rao G
10f8010519
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit dc0626b5f2
)
2020-07-31 01:30:54 -07:00
Uma Maheswara Rao G
aea1a8e2bd
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 55a2ae80dc
)
2020-07-31 01:30:30 -07:00
Uma Maheswara Rao G
35fe6fd54f
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 1f2a80b5e5
)
2020-07-31 01:29:46 -07:00
Ayush Saxena
e3b8d4eb05
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-22 18:21:14 +05:30
Chen Liang
c8c40be761
HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.
...
(cherry picked from commit 3833c616e0
)
2020-07-20 15:22:34 -07:00
Ahmed Hussein
9e7266df6c
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71
)
2020-07-15 11:40:13 -05:00
Ahmed Hussein
5969922305
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f
)
2020-07-15 09:57:36 -05:00
Shashikant Banerjee
292e22578a
HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.
...
(cherry picked from commit 82343790ee
)
2020-07-15 13:13:27 +01:00
He Xiaoqiao
e7e7a6d503
HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-13 14:31:50 +08:00
Uma Maheswara Rao G
f85ce2570e
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 3ca15292c5
)
2020-07-09 16:26:04 -07:00
Shanyu Zhao
10c9df1d0a
HDFS-15451. Do not discard non-initial block report for provided storage. ( #2119 ). Contributed by Shanyu Zhao.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-07-07 11:44:42 +08:00
Ayush Saxena
b8d9218189
HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.
2020-07-04 12:25:29 +05:30
Akira Ajisaka
3e9422d1c7
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
...
(cherry picked from commit c71ce7ac33
)
2020-06-28 16:04:47 +09:00
Virajith Jalaparti
ea97fe250c
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable ( #2100 )
...
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable
* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests
* Address Uma's comments on PR#2100
* Sort lists in test to match without concern to order
* Address comments, fix checkstyle and fix failing tests
* Fix checkstyle
(cherry picked from commit bed0a3a374
)
2020-06-27 16:22:50 -07:00
Uma Maheswara Rao G
81e33d22a0
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit d5e1bb6155
)
2020-06-27 15:42:36 -07:00
Uma Maheswara Rao G
29a8ee4be6
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 7c02d1889b
)
2020-06-27 15:42:14 -07:00
Uma Maheswara Rao G
5f67c3f3ca
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit b27810aa60
)
2020-06-27 15:41:48 -07:00
Stephen O'Donnell
7613191fcd
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 630ac9e7e5fdbc3ce86476cf0167255ab9b0470a)
2020-06-18 06:45:28 -07:00
Wei-Chiu Chuang
10880dc396
Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell."
...
This reverts commit 0b9e5ea592
.
2020-06-18 06:45:28 -07:00
S O'Donnell
3dc9db3aed
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
...
(cherry picked from commit 123777823e
)
2020-06-18 12:29:12 +01:00
Uma Maheswara Rao G
120ee793fc
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 785b1def95
)
2020-06-16 20:02:44 -07:00
Ayush Saxena
bee2846bee
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
...
(cherry picked from commit cc671b16f7
)
2020-06-16 16:54:40 -07:00
Uma Maheswara Rao G
418580446b
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 76fa0222f0
)
2020-06-16 16:54:01 -07:00
Uma Maheswara Rao G
0b5e202614
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ed83c865dd
)
2020-06-16 16:53:38 -07:00
Uma Maheswara Rao G
8e71e85af7
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 4734c77b4b
)
2020-06-16 16:53:10 -07:00
Abhishek Das
5b248de42d
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.
...
(cherry picked from commit ce4ec74453
)
2020-06-16 16:52:29 -07:00
Uma Maheswara Rao G
544996c857
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ac4a2e11d9
)
2020-06-16 16:50:57 -07:00
Stephen O'Donnell
0b9e5ea592
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 730a39d138
)
2020-06-16 16:02:07 -07:00
Takanobu Asanuma
3b8418250f
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
...
(cherry picked from commit f41a144077
)
2020-06-15 09:17:40 +09:00
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:20:10 +05:30
Ayush Saxena
043628dcf1
HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.
2020-06-10 12:09:34 +05:30
Ayush Saxena
2d4faa39e8
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:47:24 +05:30
Toshihiro Suzuki
ec8f3714e0
HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
(cherry picked from commit 545a0a147c
)
2020-06-05 11:17:13 +01:00
Wanqiang Ji
9b84a637b7
HADOOP-17055. Remove residual code of Ozone ( #2039 )
...
(cherry picked from commit d9838f2d42
)
2020-05-29 16:50:10 +09:00
Ayush Saxena
6b7040a1cb
HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
2020-05-28 13:04:22 +05:30
Ayush Saxena
718c8a5868
HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun.
2020-05-24 18:06:19 +05:30
Takanobu Asanuma
c44d8b8bdf
HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.
...
(cherry picked from commit 4d22d1c58f
)
2020-05-23 17:33:39 +09:00
Chen Liang
e452163a06
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang
...
(cherry picked from commit 7bb902bc0d
)
2020-05-19 09:52:25 +09:00
He Xiaoqiao
eb045ea056
HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1
)
2020-05-18 09:59:12 -07:00
Ayush Saxena
3915d1afc7
HDFS-15356. Unify configuration dfs.ha.allow.stale.reads
to DFSConfigKeys. Contributed by Xiaoqiao He.
...
(cherry picked from commit 178336f8a8
)
2020-05-18 09:58:59 -07:00
Wei-Chiu Chuang
240cba7e6d
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
...
(cherry picked from commit 2abcf7762a
)
2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang
289f8acc64
Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )"
...
This reverts commit ad9a6a0ee3
.
2020-05-18 09:41:03 -07:00
Wei-Chiu Chuang
53d22fdb88
Revert "HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He."
...
This reverts commit acae31aa28
.
2020-05-18 09:39:57 -07:00
Wei-Chiu Chuang
032ccba67c
HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun.
...
(cherry picked from commit 50caba1a92
)
2020-05-18 08:42:20 -07:00
He Xiaoqiao
acae31aa28
HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1
)
2020-05-18 07:10:27 -07:00
pustota2009
ad9a6a0ee3
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
...
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
(cherry picked from commit 86e6aa8eec
)
2020-05-18 07:05:03 -07:00
Surendra Singh Lilhore
7cef7109de
YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng
2020-05-14 15:17:13 +05:30
Ayush Saxena
74887ab2ad
YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng.
2020-05-14 00:37:09 +05:30
S O'Donnell
433aaeefa4
HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.
2020-05-12 15:25:05 +01:00
Ayush Saxena
ef41229f70
HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.
2020-05-12 18:43:37 +05:30
Ayush Saxena
61a4cd5539
HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.
2020-05-11 16:52:23 +05:30
Ayush Saxena
817814105a
HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.
2020-05-08 00:57:06 +05:30
Uma Maheswara Rao G
edf52d29f1
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 9c8236d04d
)
2020-05-06 15:13:33 -07:00
Ayush Saxena
4e0d99c4d6
HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.
2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. ( #1998 )
...
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53
)
2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9
HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit ebb878bab9
)
2020-05-04 10:45:18 -07:00
Ayush Saxena
df16146e7f
Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.
2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter ( #1992 )
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-02 13:59:58 -07:00
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development
2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3
HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.
...
(cherry picked from commit 9ca6298a9a
)
2020-04-29 16:01:48 +01:00
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.
...
(cherry picked from commit 5e0eda5d5f
)
2020-04-28 06:18:25 +09:00
Mingliang Liu
1db1992b31
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena
2020-04-25 15:44:01 -07:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address ( #1964 )
...
Contributed by Dhiraj Hegde.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:06:08 -07:00
Wei-Chiu Chuang
d6c48f8100
HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.
...
(cherry picked from commit 453771f170
)
2020-04-24 11:54:11 -07:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina
2020-04-22 13:41:04 -07:00
Stephen O'Donnell
aaad947c74
HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a
)
2020-04-16 20:19:00 -07:00
Wei-Chiu Chuang
749a5b81da
HDFS-15269. NameNode should check the authorization API version only … ( #1945 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 061afcdf30
)
2020-04-09 09:45:05 -07:00
Inigo Goiri
f7a94ec0a4
HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina.
2020-03-29 08:17:02 -07:00
Ayush Saxena
696a663cd0
HDFS-15193. Improving the error message for missing dfs.namenode.rpc-address.$NAMESERVICE
. Contributed by Ctest.
2020-03-29 00:01:22 +05:30
Chen Liang
f531a4a487
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.
2020-03-27 12:55:38 -07:00
Wei-Chiu Chuang
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. ( #1909 )
2020-03-25 16:03:26 -07:00
Inigo Goiri
cdcb77a2c5
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He.
2020-03-25 11:30:54 -07:00
Arpit Agarwal
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle.
...
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
2020-03-25 10:28:30 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong
2020-03-24 14:50:15 -07:00
Ayush Saxena
ea87d60493
HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun.
2020-03-24 22:56:09 +05:30
Karthik Palanisamy
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit ( #1870 )
2020-03-24 15:15:46 +05:30
He Xiaoqiao
af64ce2f4a
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 12:43:51 -07:00
Ayush Saxena
62c8015683
HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena.
2020-03-22 11:44:18 +05:30
Takanobu Asanuma
b74e47e31a
HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.
2020-03-22 01:44:55 +09:00
Ayush Saxena
f9bb2a8cc5
HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.
2020-03-19 22:26:34 +05:30
Ayush Saxena
bb41ddaf1e
HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.
2020-03-19 21:23:13 +05:30
Ayush Saxena
654db35fa2
HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
...
Contributed by Wei-Chiu Chuang.
Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
2020-03-18 13:27:13 +00:00
Ayush Saxena
8a7c54995a
HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina.
2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang
7b637a3341
HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.
2020-03-13 19:01:23 -07:00
He Xiaoqiao
e9955bb8ff
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... ( #1829 )
...
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
2020-03-13 11:29:44 -07:00
Ayush Saxena
fb1d7635ae
HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang.
2020-03-13 21:10:03 +05:30
S O'Donnell
25497799cb
HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.
2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang
20903f72b4
HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun.
2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538
HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang.
2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83
HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.
2020-03-12 18:59:17 +05:30
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
...
Contributed by Steve Loughran.
This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)
This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.
instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.
The FileSystem implementation is cut; tests updated as appropriate.
All new interfaces/classes/constants are marked as @unstable.
Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
Inigo Goiri
4062217189
HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena.
2020-03-06 10:14:53 -08:00
Inigo Goiri
3afd4cbe89
HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina.
2020-03-04 10:13:23 -08:00
Inigo Goiri
1d4d0fcbe1
HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.
2020-03-04 10:02:54 -08:00
Akshay Nehe
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. ( #1871 )
2020-03-04 11:07:31 +09:00
Ayush Saxena
1a636da041
HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun.
2020-02-29 09:55:29 +05:30
Inigo Goiri
97b797c314
HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.
2020-02-28 18:47:22 -08:00
Surendra Singh Lilhore
fccfb02672
HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.
2020-02-28 18:33:06 +05:30
Ayush Saxena
429da635ec
HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong.
2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac
HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest.
2020-02-27 22:19:35 +05:30
Surendra Singh Lilhore
2059f255d3
HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena.
2020-02-27 14:08:42 +05:30
Ayush Saxena
033a3d7ff9
HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.
2020-02-27 04:02:26 +05:30
Ayush Saxena
209630472a
HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun.
2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d
HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein.
2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990
HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang
1c5d2f1fdc
HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.
2020-02-24 13:39:16 -08:00
Ayush Saxena
7f8685f476
HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena.
2020-02-24 20:38:04 +05:30
Ayush Saxena
93b8f453b9
HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li.
2020-02-24 20:17:01 +05:30
Ayush Saxena
b5698e0c33
HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun.
2020-02-24 00:07:18 +05:30
Ayush Saxena
9eb7a8bdf8
HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi.
2020-02-23 14:25:39 +05:30
Ayush Saxena
ba9025c7cd
HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun.
2020-02-22 14:27:26 +05:30
Konstantin V Shvachko
ecbcb058b8
HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko.
2020-02-21 17:53:37 -08:00
Konstantin V Shvachko
6f84269bcd
HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko.
2020-02-21 10:51:14 -08:00
Masatake Iwasaki
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. ( #1758 )
2020-02-21 11:56:07 +09:00
Inigo Goiri
ec7507162c
HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham.
2020-02-19 11:33:58 -08:00
Ayush Saxena
ac4b556e2d
HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena.
2020-02-19 08:20:59 +05:30
Ayush Saxena
84f7638840
HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri.
2020-02-18 00:43:33 +05:30
Surendra Singh Lilhore
810783d443
HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree.
2020-02-16 12:15:03 +05:30
Ayush Saxena
c75756fe13
HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena.
2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore
a98352ced1
HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina.
2020-02-13 16:57:41 +05:30
Akira Ajisaka
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router ( #1832 )
...
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
2020-02-13 10:06:07 +09:00
Kihwal Lee
9b8a78d97b
HDFS-14758. Make lease hard limit configurable and reduce the default.
...
Contributed by hemanthboyina.
2020-02-11 12:40:00 -06:00
Stephen O'Donnell
d7c136b9ed
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-11 08:00:15 -08:00
Ayush Saxena
6191d4b4a0
HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun.
2020-02-09 23:32:22 +05:30
Ayush Saxena
d23317b102
HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang
2020-02-08 10:33:57 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
Chen Liang
ce7b8b5634
HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.
2020-02-04 12:12:35 -08:00
Kihwal Lee
10a60fbe20
HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan.
2020-02-04 12:22:35 -06:00
Stephen O'Donnell
1e3a0b0d93
HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2020-01-31 16:13:02 -08:00
Kihwal Lee
799d4c1cf4
HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails
...
intermittently. Contributed by Ahmed Hussein.
2020-01-29 11:00:27 -06:00
Chen Liang
483397c7f7
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang
2020-01-28 15:20:36 -08:00
Chen Liang
3e86807802
Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang."
...
This reverts commit ff8ff0f7e5
.
2020-01-28 15:19:47 -08:00
Chen Liang
ff8ff0f7e5
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang.
2020-01-28 15:14:58 -08:00
Inigo Goiri
1839c467f6
HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein.
2020-01-28 10:10:35 -08:00
Ayush Saxena
87c198468b
HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun.
2020-01-28 23:09:38 +05:30
Ayush Saxena
f876dc228b
HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena.
2020-01-28 19:34:21 +05:30
Ayush Saxena
6d008c0d39
HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest
2020-01-25 11:47:29 +05:30
Kihwal Lee
d10f77e3c9
HDFS-15119. Allow expiration of cached locations in DFSInputStream.
...
Contributed by Ahmed Hussein.
2020-01-24 09:15:27 -06:00
Ayush Saxena
92c58901d7
HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena
2020-01-23 18:18:34 +05:30
Kihwal Lee
bd03053ea2
HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein.
2020-01-22 09:14:38 -06:00
Inigo Goiri
8cfc3673dc
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui.
2020-01-21 13:29:20 -08:00
Inigo Goiri
b657822b98
HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein.
2020-01-21 13:22:53 -08:00
Akira Ajisaka
1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.
2020-01-21 10:58:32 +09:00
Chao Sun
d7c4f8ab21
HDFS-13616. Batch listing of multiple directories ( #1725 )
2020-01-15 17:22:47 -08:00
Vinayakumar B
52b360a928
HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. ( #1801 )
2020-01-13 10:40:29 +05:30
Ayush Saxena
cebce0a348
HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena.
2020-01-11 10:29:11 +05:30
Chen Liang
26a969ec73
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.
2020-01-10 17:06:59 -08:00
Kihwal Lee
5fb901ac40
HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein.
2020-01-10 12:43:21 -06:00
Ayush Saxena
b32757c616
HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena.
2020-01-10 08:14:56 +05:30
Surendra Singh Lilhore
bf45f3b80a
HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina.
2020-01-09 12:04:05 +05:30
Rakesh Radhakrishnan
7030722e5d
HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.
2020-01-08 14:25:17 +05:30
Masatake Iwasaki
aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. ( #1797 )
2020-01-08 16:45:39 +09:00
Masatake Iwasaki
a43c177f1d
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. ( #1775 )
2020-01-08 11:28:34 +09:00
Ayush Saxena
f8644fbe9f
HDFS-15091. Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock. Contributed by Ayush Saxena.
2020-01-04 19:02:59 +05:30
Masatake Iwasaki
037ec8cfb1
HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2020-01-04 01:55:36 +09:00
Rakesh Radhakrishnan
d79cce20ab
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.
2020-01-02 11:44:00 +05:30
Takanobu Asanuma
074050ca59
HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina.
2020-01-01 11:26:38 +09:00
Surendra Singh Lilhore
ee51eadda0
HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina.
2019-12-29 11:15:54 +05:30
Takanobu Asanuma
dc32f583af
HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena.
2019-12-28 21:32:15 +09:00
Ayush Saxena
8730a7bf60
HDFS-15003. RBF: Make Router support storage type quota. Contributed by Jinglun.
2019-12-27 09:22:58 +05:30
Ayush Saxena
80f91d14ab
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He.
2019-12-27 09:12:44 +05:30
Ayush Saxena
300505c562
HDFS-15054. Delete Snapshot not updating new modification time. Contributed by hemanthboyina.
2019-12-25 12:42:50 +05:30
Konstantin V Shvachko
b98ac2a3af
HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko.
2019-12-24 12:08:34 -08:00
Inigo Goiri
52d7b745c6
HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui.
2019-12-19 09:42:17 -08:00
Inigo Goiri
b86895485d
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He.
2019-12-19 09:34:43 -08:00
Shashikant Banerjee
fdd96e46d1
HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee.
2019-12-18 22:50:46 +05:30
Inigo Goiri
24080666e5
HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun.
2019-12-16 18:41:45 -08:00
Masatake Iwasaki
dc6cf17b34
HDFS-15048. Fix findbug in DirectoryScanner.
2019-12-16 19:00:57 +09:00
Ayush Saxena
7a87007545
HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina.
2019-12-15 00:33:32 +05:30
Chen Liang
65c4660bcd
HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang.
2019-12-12 10:22:05 -08:00
Masatake Iwasaki
bdd00f10b4
HDFS-15047. Document the new decommission monitor (HDFS-14854). ( #1755 )
2019-12-13 00:35:41 +09:00
Takanobu Asanuma
93bb368094
HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li
2019-12-12 10:56:19 +09:00
Erik Krogen
c174d50b30
HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen.
2019-12-11 09:39:37 -08:00
Stephen O'Donnell
c93cb6790e
HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-10 17:16:43 -08:00
Masatake Iwasaki
11cd5b6e39
HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-12-07 08:46:45 +09:00
Wei-Chiu Chuang
ecd461f940
HDFS-14751. Synchronize on diffs in DirectoryScanner. Contributed by Lisheng Sun.
2019-12-06 15:10:40 -08:00
Wei-Chiu Chuang
313b76f8e9
HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow.
2019-12-06 15:10:40 -08:00
Ayush Saxena
705b172b95
HDFS-14998. [SBN read] Update Observer Namenode doc for ZKFC after HDFS-14130. Contributed by Fei Hui.
2019-12-06 23:04:09 +05:30
Ayush Saxena
83a14559e5
HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui.
2019-12-05 18:52:01 +05:30
Ayush Saxena
0c217feed8
HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun.
2019-12-03 09:18:09 +05:30
Ayush Saxena
6b2d6d4aaf
HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina.
2019-11-29 23:55:37 +05:30
Surendra Singh Lilhore
44f7b9159d
HDFS-15013. Reduce NameNode overview tab response time. Contributed by HuangTao.
2019-11-29 00:42:31 +05:30
Surendra Singh Lilhore
0384687811
HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore.
2019-11-28 23:49:35 +05:30
Ayush Saxena
46166bd8d1
HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena.
2019-11-28 18:28:20 +05:30
Yiqun Lin
c3659f8f94
HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun.
2019-11-28 14:41:49 +08:00
Yiqun Lin
2b452b4e60
HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
2019-11-28 10:43:35 +08:00
Yiqun Lin
c8bef4d6a6
HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun.
2019-11-27 10:57:20 +08:00
Ayush Saxena
049940e77b
HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar.
2019-11-23 02:30:02 +05:30
Takanobu Asanuma
b25e94ce29
HDFS-14924. RenameSnapshot not updating new modification time. Contributed by hemanthboyina
2019-11-22 13:48:02 +09:00
Yiqun Lin
9b6906fe91
HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun.
2019-11-22 10:53:55 +08:00
Surendra Singh Lilhore
26270196a2
HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina.
2019-11-22 00:31:25 +05:30
Ayush Saxena
98d249dcda
HDFS-14996. RBF: GetFileStatus fails for directory with EC policy set in case of multiple destinations. Contributed by Ayush Saxena.
2019-11-21 12:08:42 +05:30
Surendra Singh Lilhore
fd264b8265
HDFS-14995. Use log variable directly instead of passing as argument in InvalidateBlocks#printBlockDeletionTime(). Contributed by Lisheng Sun.
2019-11-21 00:00:42 +05:30
He Xiaoqiao
0b50aa29fd
HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
2019-11-19 12:16:57 -08:00
Surendra Singh Lilhore
c870514740
HDFS-14992. TestOfflineEditsViewer is failing in Trunk. Contributed by hemanthboyina.
2019-11-20 00:07:43 +05:30
Ayush Saxena
34cb595d2e
HDFS-14967. TestWebHDFS fails in Windows. Contributed by Renukaprasad C.
2019-11-18 18:49:58 +05:30
Yiqun Lin
b3119b9ab6
HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun.
2019-11-16 11:32:41 +08:00
Hui Fei
67f2c491fe
HDFS-14802. The feature of protect directories should be used in RenameOp ( #1669 )
2019-11-15 12:35:37 -08:00
He Xiaoqiao
c892a879dd
HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
2019-11-15 12:16:31 -08:00
Erik Krogen
b2cc8b6b4a
HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen.
2019-11-15 10:10:37 -08:00
Erik Krogen
586defe711
HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen.
2019-11-13 11:46:35 -08:00
Chao Sun
97ec34e117
HDFS-14959: [SBNN read] access time should be turned off ( #1706 ) Contributed by Chao Sun.
2019-11-12 11:35:50 -08:00
Inigo Goiri
40150da1e1
HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina.
2019-11-12 10:53:54 -08:00
Ayush Saxena
77934bc07b
HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui.
2019-11-11 11:05:34 +05:30
Takanobu Asanuma
6663d6a5c2
HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li
2019-11-11 14:04:57 +09:00
Surendra Singh Lilhore
320008bb7c
HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina.
2019-11-09 22:14:35 +05:30
Ayush Saxena
247584eb63
HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan.
2019-11-07 22:49:01 +05:30
Chen Liang
dd900259c4
HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko.
2019-11-06 09:56:19 -08:00
Chen Liang
9d0d580031
HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen.
2019-11-06 09:30:04 -08:00
Ayush Saxena
9e287054a8
HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun.
2019-11-06 22:02:03 +05:30
Ayush Saxena
dcf55838ae
HDFS-14880. Correct the sequence of statistics & exit message in balencer. Contributed by Renukaprasad C.
2019-11-06 21:45:03 +05:30
Surendra Singh Lilhore
c36014165c
HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore.
2019-11-06 19:28:55 +05:30
Inigo Goiri
bfb8f28cc9
HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang.
2019-11-05 09:19:52 -08:00
Ayush Saxena
b643a1cbe8
HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun.
2019-11-05 09:23:20 +05:30
Ayush Saxena
2ffec347eb
HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui.
2019-11-05 01:37:15 +05:30
Wei-Chiu Chuang
eb73ba6ed5
HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao.
2019-11-04 09:40:42 -08:00
Zhao Junwang
de6b8b0c0b
HDFS-14925. Rename operation should check nest snapshot ( #1670 )
...
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.
Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
2019-11-01 16:37:53 -07:00
Ayush Saxena
7d7acb004a
HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong.
2019-11-01 22:50:10 +05:30
Surendra Singh Lilhore
02009c3bb7
HDFS-14768. EC : Busy DN replica should be consider in live replica check. Contributed by guojh.
2019-11-01 22:15:05 +05:30
Ayush Saxena
9d25ae7669
HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui.
2019-10-31 23:49:37 +05:30
Ayush Saxena
d9fbedc4ae
HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun.
2019-10-31 23:39:27 +05:30
Jonathan Hung
e6137d0e50
Make upstream aware of 2.10.0 release
2019-10-30 20:58:32 -07:00
Ayush Saxena
fa4904cdca
HDFS-14935. Refactor DFSNetworkTopology#isNodeInScope. Contributed by Lisheng Sun.
2019-10-29 12:30:24 +05:30
Eric Yang
30ed24a421
HDFS-14730. Removed unused configuration dfs.web.authentication.filter.
...
Contributed by Chen Zhang
2019-10-28 19:27:32 -04:00
Eric Badger
9ef6ed9c1c
HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger
2019-10-28 16:37:04 +00:00
zhaoyim
30db895b59
HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() ( #1667 )
...
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-25 13:09:14 -07:00
Takanobu Asanuma
862526530a
HDFS-14933. Fixing a typo in documentation of Observer NameNode. Contributed by Xieming Li
2019-10-25 15:23:05 +09:00
Takanobu Asanuma
0db0f1e399
HDFS-14917. Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html". Contributed by Xieming Li
2019-10-25 10:27:38 +09:00
Wei-Chiu Chuang
a1b4eebcc9
HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. ( #1672 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2019-10-24 13:09:52 -07:00
Eric Yang
ac6b6a6a85
HADOOP-16614. Add aarch64 support for dependent leveldbjni.
...
Contributed by liusheng
close #1546
2019-10-24 11:45:57 -04:00
Vinayakumar B
ee699dc26c
HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena.
2019-10-24 12:14:09 +05:30
Mukul Kumar Singh
a901405ad8
HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-22 13:31:15 -07:00
Ayush Saxena
6020505943
HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena.
2019-10-22 18:44:22 +05:30
Ayush Saxena
19f35cfd57
HDFS-14918. Remove useless getRedundancyThread from BlockManagerTestUtil. Contributed by Fei Hui.
2019-10-22 18:34:02 +05:30
Wei-Chiu Chuang
72003b19bf
HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun.
2019-10-21 17:31:43 -07:00
Kihwal Lee
c4e27ef773
HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He.
2019-10-21 16:33:15 -05:00