Viraj Jasani
278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster ( #4028 ). Contributed by Viraj Jasani.
2022-03-04 22:17:48 +05:30
Chao Sun
f800b65b40
Make upstream aware of 3.3.2 release
2022-03-02 19:14:50 -08:00
litao
9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode ( #4041 )
2022-03-02 11:25:50 +08:00
SevenAddSix
5eab9719cb
HDFS-16480. Fix typo: indicies -> indices ( #4020 )
2022-02-28 21:12:18 +08:00
Hui Fei
b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode ( #3998 )" ( #4038 )
2022-02-28 21:05:19 +08:00
litao
8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode ( #3998 )
2022-02-28 21:03:59 +08:00
litao
6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode ( #3828 )
2022-02-25 01:23:20 +09:00
Viraj Jasani
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics ( #4002 )
2022-02-22 13:04:36 +09:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. ( #3861 )
2022-02-22 10:15:19 +08:00
Chentao Yu
19d90e62fb
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953)
2022-02-15 15:58:24 -08:00
litao
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode ( #3827 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2022-02-15 13:40:46 +09:00
secfree
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads ( #3847 )
2022-02-09 10:07:05 +08:00
singer-bin
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … ( #3926 )
2022-02-06 13:05:57 +08:00
KevinWikant
089e06de21
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception ( #3942 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-31 13:04:32 +09:00
litao
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web ( #3943 )
2022-01-30 16:13:58 +08:00
secfree
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure ( #3850 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 17:04:11 +09:00
Li MingXiang
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. ( #3900 ). Contributed by limingxiang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-27 16:53:21 +08:00
litao
6136d630a3
HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped ( #3888 )
2022-01-27 13:42:44 +09:00
litao
c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode ( #3831 )
2022-01-26 17:24:09 +09:00
jianghuazhu
8c7c49d877
HDFS-16401.Remove the worthless DatasetVolumeChecker#numAsyncDatasetChecks. ( #3838 )
2022-01-25 21:51:17 +08:00
Bryan Beaudreault
94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream ( #3527 )
2022-01-25 11:42:35 +00:00
Thinker313
43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename ( #3898 ). Contributed by lei w.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-25 15:26:18 +08:00
daimin
5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy ( #3899 ). Contributed by daimin.
...
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-01-24 12:04:26 +05:30
litao
15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. ( #3839 ). Contributed by tomscut.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-24 14:26:30 +08:00
qinyuren
db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages ( #3883 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-19 17:59:42 +09:00
litao
e355646330
HDFS-16399. Reconfig cache report parameters for datanode ( #3841 )
2022-01-19 15:10:39 +09:00
qinyuren
fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force ( #3887 )
2022-01-19 13:42:33 +09:00
Jackson Wang
d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. ( #3880 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-14 22:38:11 +09:00
litao
f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode ( #3843 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-14 14:48:10 +09:00
Xiangyi Zhu
034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously ( #3063 ). Contributed by Xiangyi Zhu.
...
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-13 12:56:19 +08:00
luoyuan3471
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
litao
46b02788bd
HDFS-16371. Exclude slow disks when choosing volume ( #3753 )
2022-01-06 09:31:00 +09:00
Ashutosh Gupta
43e5218a86
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader ( #3854 )
...
Contributed by Ashutosh Gupta
2022-01-05 15:51:14 +00:00
liever18
e1d0aa9ee5
HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero ( #3856 )
2022-01-05 15:42:58 +00:00
Ashutosh Gupta
9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue ( #3835 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-04 13:25:16 +09:00
Symious
b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline ( #3704 )
2021-12-30 00:27:11 +09:00
jianghuazhu
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). ( #3823 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 17:24:12 +09:00
KevinWikant
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning ( #3675 )
...
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 22:59:08 +09:00
secfree
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout ( #3821 )
2021-12-22 10:44:07 +08:00
Jackson Wang
9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. ( #3803 )
...
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
2021-12-21 11:04:02 +09:00
jianghuazhu
746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. ( #3806 )
2021-12-20 11:28:55 +00:00
secfree
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout ( #3815 )
2021-12-20 18:16:22 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
liubingxing
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport ( #3714 ). Contributed by liubingxing.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-17 13:31:28 +08:00
litao
22f5e1885d
HDFS-16377. Should CheckNotNull before access FsDatasetSpi ( #3784 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-16 13:49:50 +09:00
litao
10cdde59b4
HDFS-16375. The FBR lease ID should be exposed to the log ( #3769 )
2021-12-16 13:29:32 +09:00
litao
59c650802d
HDFS-16378. Add datanode address to BlockReportLeaseManager logs ( #3786 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-15 11:16:32 +08:00
Ayush Saxena
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. ( #3756 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-14 17:21:51 +09:00
litao
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable ( #3716 )
2021-12-14 11:19:06 +09:00
Akira Ajisaka
9b9e2ef87f
HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure ( #3774 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-12-10 01:36:31 +09:00
qinyuren
35556ea043
HDFS-16333. fix balancer bug when transfer an EC block ( #3679 )
2021-12-09 13:24:09 +09:00
PHILO-HE
1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support ( #3761 )
2021-12-08 17:52:40 +05:30
GuoPhilipse
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode ( #3661 )
...
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 21:21:48 +09:00
GuoPhilipse
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy ( #3696 ). Contributed by guophilipse.
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-08 16:34:54 +05:30
Masatake Iwasaki
e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. ( #3740 )
2021-12-07 21:39:04 +09:00
GuoPhilipse
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem ( #3713 ). Contributed by guophilipse.
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-07 14:19:27 +05:30
GuoPhilipse
1509ea5de2
HDFS-16338. Correct fsimage error configuration message ( #3684 ). Contributed by guophilipse.
...
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-06 14:00:50 +05:30
Takanobu Asanuma
e8e69de106
HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu.
2021-12-06 10:44:36 +09:00
litao
c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo ( #3747 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-04 22:54:44 +05:30
huhaiyang
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable ( #3664 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-04 02:19:36 +09:00
bitterfox
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake ( #3677 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-03 23:30:13 +09:00
wangzhaohui
0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 ( #3742 )
2021-12-03 16:51:04 +05:30
litao
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable ( #3676 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-03 14:12:05 +09:00
litao
1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand ( #3732 ). Contributed by tomscut.
2021-12-01 12:48:21 +05:30
litao
829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString ( #3695 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-29 17:48:10 +09:00
litao
86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded ( #3689 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-26 19:52:14 +09:00
Takanobu Asanuma
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. ( #3698 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-25 14:05:04 +09:00
Viraj Jasani
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully ( #3711 )
2021-11-25 10:23:04 +08:00
huhaiyang
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3708 )
2021-11-25 10:20:42 +08:00
Viraj Jasani
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback ( #3686 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 22:54:11 +09:00
Mukul Kumar Singh
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. ( #3694 )
2021-11-23 13:36:14 +00:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web ( #3682 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode ( #3654 )
2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md ( #3681 )
2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )" ( #3697 )
2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description ( #3678 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer ( #3663 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc ( #3669 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer ( #3672 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager ( #3670 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router ( #3635 )
2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )
2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers ( #3659 )
2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. ( #3320 )
2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode ( #3643 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms ( #3648 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-11-15 22:57:24 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP ( #3655 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable ( #3596 )
2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. ( #3629 )
2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics ( #3636 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-12 10:23:25 +09:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents ( #3637 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:22:32 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. ( #3622 )
2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics ( #3616 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability
2021-11-08 10:21:39 -08:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. ( #3605 )
2021-11-04 16:43:25 +09:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. ( #3602 )
2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log ( #3538 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file ( #3593 )
2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) ( #3598 )
2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. ( #3544 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-02 00:55:50 +09:00
jianghuazhu
0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). ( #3600 )
...
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600 ). Contributed by JiangHua Zhu.
2021-10-31 08:48:52 +05:30
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. ( #3374 )
2021-10-31 09:29:34 +09:00
litao
6aa664c624
HDFS-16279. Print detail datanode info when process first storage report ( #3564 )
2021-10-29 10:27:00 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout ( #3574 )
2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules ( #3566 )
2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin."
...
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc
.
2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:09:37 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy ( #3559 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a
HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker ( #3571 )
2021-10-21 13:41:46 -07:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery ( #3548 )
2021-10-18 11:55:55 +01:00
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt ( #3554 )
...
* HDFS-16274. Improve log for FSNamesystem#startFileInt
* fix failed unit test
2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE ( #3546 )
2021-10-13 18:14:03 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). ( #3497 )
2021-10-13 17:21:03 +08:00
jianghuazhu
d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. ( #3533 )
2021-10-11 19:49:58 +08:00
langlaile1221
88d8c3d0c8
HDFS-16241. Standby close reconstruction thread ( #3493 )
...
Co-authored-by: zhanghuazong <Zhz@20068867>
2021-10-11 16:11:31 +08:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules ( #3530 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-11 15:32:49 +09:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
S O'Donnell
a30231ff8e
HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell
2021-10-06 15:04:04 +01:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. ( #3422 ). Contributed by Ayush Saxena.
...
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. ( #3494 )
...
Contributed by Max Xie
2021-10-01 09:35:14 +02:00
jianghuazhu
1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. ( #3483 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-10-01 10:30:41 +08:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics ( #3471 )
2021-09-28 13:09:18 +08:00
jianghuazhu
02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. ( #3479 )
2021-09-28 13:07:36 +08:00
Chao Sun
1397cf2496
Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same"
...
This reverts commit 59f01a548e
.
2021-09-25 21:32:43 -07:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. ( #3468 )
2021-09-24 08:35:23 -07:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov.
2021-09-23 10:45:02 +05:30
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 ( #3427 )
2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink ( #3386 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:11:20 +09:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. ( #3455 ). Contributed by Thomas Leplus.
2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile ( #3429 )
2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. ( #3451 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-18 11:06:51 +05:30
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out ( #3425 )
2021-09-16 23:59:36 -07:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation ( #3366 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-09-16 10:00:02 +09:00
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount ( #3317 )
...
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
2021-09-15 09:56:51 -07:00
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. ( #2831 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown ( #3359 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-09-14 13:18:15 +08:00
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. ( #3404 ). Contributed by Ayush Saxena.
2021-09-13 22:50:39 +05:30
Ayush Saxena
17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. ( #3424 ). Contributed by Ayush Saxena.
2021-09-13 17:54:38 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing ( #3340 )
2021-09-13 10:25:19 +05:30
jianghuazhu
edfde6eebc
HDFS-16197.Simplify getting NNStorage in FSNamesystem. ( #3357 )
2021-09-12 19:58:30 +08:00
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router ( #3421 )
2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS ( #3346 ) Contributed by Leon Gao
...
* Router to support resolving monitored namenodes with DNS
* Style
* fix style and test failure
* Add test for NNHAServiceTarget const
* Resolve comments
* Fix test
* Comments and style
* Create a simple function to extract port
* Use LambdaTestUtils.intercept
* fix javadoc
* Trigger Build
2021-09-10 16:40:08 -07:00
Ahmed Hussein
bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query ( #3375 )
2021-09-08 21:21:16 -07:00
litao
5a306955a4
HDFS-16209. Add description for dfs.namenode.caching.enabled ( #3378 )
2021-09-08 12:38:55 +08:00
jianghuazhu
4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. ( #3368 )
2021-09-06 18:06:12 +08:00
Renukaprasad C
dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack ( #3244 ). Contributed by Renukaprasad C.
2021-09-05 17:07:59 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname ( #3354 ). Contributed by tomscut.
2021-09-05 14:19:01 +08:00
langlaile1221
24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue ( #2894 )
2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. ( #3353 )
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2021-08-31 12:25:03 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits ( #3235 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. ( #3284 )
...
* Add DNS resolution for QJM
* Add log
* Resolve comments
* checkstyle
* typo
2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile ( #3329 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-08-25 17:37:25 +09:00
Neil
9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. ( #3315 )
2021-08-24 11:15:47 +08:00
litao
07627ef19e
HDFS-16177. Bug fix for Util#receiveFile ( #3310 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-08-19 12:30:38 +08:00
litao
0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat ( #3291 )
2021-08-19 09:51:00 +08:00
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. ( #3295 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-08-17 16:47:10 +08:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus ( #3280 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map ( #3296 )
2021-08-16 13:40:05 +08:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely ( #3270 )
...
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-08-06 18:23:00 +08:00
wangzhaohui
a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager ( #3269 ). Contributed by wangzhaohui.
2021-08-06 09:27:05 +01:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… ( #3247 ) Contributed by Shuyan Zhang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-08-04 00:22:21 +08:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 ( #3241 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found ( #3243 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-30 08:29:56 -07:00
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. ( #2836 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
2021-07-29 11:19:17 +08:00
S O'Donnell
4eae284827
HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell
2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. ( #3175 )
...
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a
Co-authored-by: Zhihai Xu <zxu@apache.org>
2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. ( #3185 )
...
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically ( #3228 ). Contributed by Viraj Jasani.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. ( #3229 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web ( #3211 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-26 12:45:01 +08:00
Akira Ajisaka
2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. ( #3227 )
...
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"
This reverts commit b6d839a60c
.
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
2021-07-26 01:53:39 +09:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.
2021-07-17 21:57:00 +08:00
crossfire
4bb25c810b
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable ( #2414 )
2021-07-15 19:58:12 -07:00
Abhishek Das
1dd03cc4b5
HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das ( #2260 )
2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. ( #2639 )
...
* Rebase trunk
* Fix to use FQDN and update config name
* Fix javac
* Style and trigger build
* Trigger Build after force push
* Trigger Build
* Fix config names
2021-07-13 14:18:59 -07:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
S O'Donnell
b6c06c4b76
HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma
2021-07-09 10:03:57 +01:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. ( #3188 )
2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… ( #3140 )
2021-07-08 14:10:45 +08:00
JiaguodongF
9381071758
HDFS-16114. the balancer parameters print error ( #3179 )
...
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-07 14:13:05 +05:30
Ayush Saxena
c85a18327d
HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w.
2021-07-06 23:26:52 +05:30
Viraj Jasani
da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer ( #3169 )
2021-07-05 09:23:24 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout ( #3172 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-05 08:14:37 +09:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell ( #3168 )
2021-07-02 15:31:25 +08:00
zhangshuyan0
9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. ( #3159 )
...
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
2021-07-01 23:52:21 +08:00
litao
52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI ( #3160 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-01 18:44:10 +05:30
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing ( #3136 )
2021-06-30 22:06:56 -07:00
Viraj Jasani
d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts ( #3148 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack Chuang <stack@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-30 12:55:07 -07:00
Xiangyi Zhu
4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota ( #3156 )
...
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-06-30 00:12:08 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor ( #3146 )
2021-06-29 03:15:12 -07:00
Viraj Jasani
93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects ( #3150 )
2021-06-29 02:31:02 -07:00
litao
f78b6d47f9
HDFS-16085. Move the getPermissionChecker out of the read lock ( #3134 ). Contributed by tomscut.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-24 15:59:42 +05:30
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations ( #3117 )
2021-06-24 11:27:40 +09:00
Viraj Jasani
d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer ( #3127 )
2021-06-22 20:21:34 -07:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-06-21 10:25:12 +09:00