big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,958 Commits 2 Branches 2 Tags 264 MiB
882f08b4bc
Commit Graph

5454 Commits

Author SHA1 Message Date
litao
9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041) 2022-03-02 11:25:50 +08:00
SevenAddSix
5eab9719cb
HDFS-16480. Fix typo: indicies -> indices (#4020) 2022-02-28 21:12:18 +08:00
Hui Fei
b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998)" (#4038) 2022-02-28 21:05:19 +08:00
litao
8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998) 2022-02-28 21:03:59 +08:00
litao
6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 2022-02-25 01:23:20 +09:00
Viraj Jasani
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Chentao Yu
19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
litao
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 13:40:46 +09:00
secfree
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847) 2022-02-09 10:07:05 +08:00
singer-bin
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 2022-02-06 13:05:57 +08:00
litao
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web (#3943) 2022-01-30 16:13:58 +08:00
secfree
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 17:04:11 +09:00
Li MingXiang
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00
litao
c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode (#3831) 2022-01-26 17:24:09 +09:00
Bryan Beaudreault
94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 2022-01-25 11:42:35 +00:00
Thinker313
43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-25 15:26:18 +08:00
daimin
5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-01-24 12:04:26 +05:30
litao
15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. (#3839). Contributed by tomscut. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-24 14:26:30 +08:00
qinyuren
db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages (#3883) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-19 17:59:42 +09:00
litao
e355646330
HDFS-16399. Reconfig cache report parameters for datanode (#3841) 2022-01-19 15:10:39 +09:00
qinyuren
fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 2022-01-19 13:42:33 +09:00
Jackson Wang
d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 22:38:11 +09:00
litao
f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 14:48:10 +09:00
Xiangyi Zhu
034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063). Contributed by Xiangyi Zhu. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-13 12:56:19 +08:00
litao
46b02788bd
HDFS-16371. Exclude slow disks when choosing volume (#3753) 2022-01-06 09:31:00 +09:00
Ashutosh Gupta
9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 13:25:16 +09:00
Symious
b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline (#3704) 2021-12-30 00:27:11 +09:00
jianghuazhu
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 17:24:12 +09:00
KevinWikant
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 22:59:08 +09:00
secfree
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
secfree
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
liubingxing
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-17 13:31:28 +08:00
Ayush Saxena
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:21:51 +09:00
litao
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 2021-12-14 11:19:06 +09:00
qinyuren
35556ea043
HDFS-16333. fix balancer bug when transfer an EC block (#3679) 2021-12-09 13:24:09 +09:00
GuoPhilipse
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode (#3661) 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 21:21:48 +09:00
GuoPhilipse
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-08 16:34:54 +05:30
GuoPhilipse
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem (#3713). Contributed by guophilipse. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-07 14:19:27 +05:30
GuoPhilipse
1509ea5de2
HDFS-16338. Correct fsimage error configuration message (#3684). Contributed by guophilipse. 
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-06 14:00:50 +05:30
Takanobu Asanuma
e8e69de106 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 2021-12-06 10:44:36 +09:00
huhaiyang
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-04 02:19:36 +09:00
bitterfox
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-03 23:30:13 +09:00
litao
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-03 14:12:05 +09:00
Takanobu Asanuma
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-25 14:05:04 +09:00
Viraj Jasani
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 2021-11-25 10:23:04 +08:00
huhaiyang
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Viraj Jasani
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 22:54:11 +09:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-09 17:06:28 +09:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-02 00:55:50 +09:00
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554) 
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
 2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 15:32:49 +09:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494) 
Contributed by Max Xie
 2021-10-01 09:35:14 +02:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471) 2021-09-28 13:09:18 +08:00
Chao Sun
1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" 
This reverts commit 59f01a548e.
 2021-09-25 21:32:43 -07:00
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:11:20 +09:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-09-14 13:18:15 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) 2021-09-13 10:25:19 +05:30
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Ahmed Hussein
bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 2021-09-08 21:21:16 -07:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2021-08-31 12:25:03 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
S O'Donnell
4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175) 
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
 2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185) 
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-26 12:45:01 +08:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00
Abhishek Das
1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-05 08:14:37 +09:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
Ahmed Hussein
c255feb5d7
HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:22:47 +05:30
LeonGao
643dfd60e4
HDFS-15842. HDFS mover to emit metrics. (#2738) 2021-06-19 15:39:46 -07:00
hust_hhb
56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065) 2021-06-17 11:08:29 +08:00
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani
 2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
Ahmed Hussein
352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097) 2021-06-11 11:00:42 -05:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-11 17:30:55 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Konstantin V Shvachko
8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko
1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
Viraj Jasani
acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao
85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Viraj Jasani
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
Brahma Reddy Battula
ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula
0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
Ayush Saxena
cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
touchida
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Viraj Jasani
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:47:45 -07:00
Viraj Jasani
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-19 20:37:44 -07:00
touchida
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-17 12:00:05 +09:00
Hui Fei
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 2021-03-17 09:11:30 +08:00
sunlisheng
7025f39944 HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 2021-03-15 11:34:13 +08:00
He Xiaoqiao
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun. 2021-03-14 17:48:18 +08:00
Xiaoyu Yao
7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764) 2021-03-12 11:27:53 -08:00
Hui Fei
6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate (#2746) 2021-03-10 14:11:29 +08:00
litao
cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
 2021-03-03 22:34:02 +09:00
Konstantin V Shvachko
9501c698f4 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 2021-03-01 17:17:23 -08:00
S O'Donnell
c3b3b36dee HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 2021-03-01 11:36:41 +00:00
LeonGao
940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704) 2021-02-23 22:29:48 -08:00
Ayush Saxena
ff84a57483 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 2021-02-20 19:48:00 +05:30
Hui Fei
2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:07:22 +08:00
Ayush Saxena
e391844e8e HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. 2021-02-18 01:39:04 +05:30
Akira Ajisaka
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Zehao Chen
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
LeonGao
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 2021-02-09 10:49:28 +00:00
bshashikant
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. (#2677) 2021-02-09 10:27:51 +05:30
LeonGao
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625) 2021-02-08 11:01:21 -08:00
bshashikant
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682) 2021-02-06 08:26:12 -08:00
Jim Brennan
c4918fb298 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 21:20:30 +00:00
maobaolong
b99dba38b9
HDFS-15819. Fix import style #2681 2021-02-05 10:11:27 +08:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
crossfire
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 2021-02-02 09:02:09 +00:00
He Xiaoqiao
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:50:54 +08:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
He Xiaoqiao
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:28:29 +08:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
Jim Brennan
6184ab07f6 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 2021-01-28 16:47:52 +00:00
sunlisheng
f8769e0f4b HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 2021-01-28 10:10:39 +08:00
He Xiaoqiao
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Masatake Iwasaki
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-14 16:19:35 +09:00
touchida
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-13 11:23:07 +09:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
Masatake Iwasaki
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. (#2501) 2021-01-04 14:00:57 +09:00
Ayush Saxena
ba496f61f8 HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina. 2020-12-22 02:16:00 +05:30
Ayush Saxena
2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ahmed Hussein
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-17 11:33:29 -06:00
Ayush Saxena
0da00fc743 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-14 20:42:23 -08:00
S O'Donnell
9ed737001c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-11 20:53:10 +00:00
Ahmed Hussein
be35fa186c
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 11:02:42 -06:00
Ahmed Hussein
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 2020-12-08 13:25:24 -06:00
Ahmed Hussein
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. (#2516) 2020-12-07 14:22:06 -06:00
Ayush Saxena
ad40715690 HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun. 2020-12-07 15:00:19 +05:30
Hui Fei
e2c1268ebd HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 09:20:09 +08:00
LeonGao
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483). Contributed by Leon Gao. 2020-12-02 15:53:09 -08:00
Ahmed Hussein
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 15:38:20 -06:00
Ahmed Hussein
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 13:59:00 -06:00
Ahmed Hussein
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 2020-12-01 15:02:53 -06:00
Siyao Meng
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472) 2020-11-25 11:01:04 -08:00
He Xiaoqiao
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2020-11-23 11:26:52 +08:00
Ahmed Hussein
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2020-11-20 18:03:45 +05:30
Masatake Iwasaki
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2020-11-18 16:11:09 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
maobaolong
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 2020-11-10 13:01:10 +08:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
He Xiaoqiao
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
 2020-11-04 13:53:46 +08:00
Konstantin V Shvachko
d07dc7afb4 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 2020-11-03 12:02:40 -08:00
Ahmed Hussein
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2020-10-28 16:24:34 -07:00
Ahmed Hussein
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406) 2020-10-27 18:56:40 -07:00
Ahmed Hussein
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2020-10-27 18:52:56 -07:00
Hui Fei
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 2020-10-28 09:13:25 +08:00
Ahmed Hussein
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 20:45:00 +09:00
Akira Ajisaka
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309) 2020-10-27 19:18:08 +09:00
Ahmed Hussein
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 16:56:30 +09:00
Hui Fei
12c908c827 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 2020-10-26 22:12:27 +08:00
Inigo Goiri
e7aa4da0a2 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 2020-10-23 09:47:10 -07:00
Kihwal Lee
da1b6e3cc2 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 2020-10-22 16:59:49 -05:00
Kihwal Lee
88a9f42f32 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-21 00:59:35 -05:00
Ayush Saxena
2e8cafac3b HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 2020-10-19 10:48:47 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Siyao Meng
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370) 2020-10-13 10:59:42 -07:00
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same 
Contributed by Gabor Bota.
 2020-10-13 16:17:44 +01:00
Mukund Thakur
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:31:00 +01:00
Steve Loughran
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380)" 
(reverting to change the JIRA number)

This reverts commit 3e1b1dfd77.

Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd
 2020-10-13 13:30:38 +01:00
Mukund Thakur
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.
 2020-10-13 13:27:55 +01:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378). Contributed by Uma Maheswara Rao G. 2020-10-12 14:22:44 -07:00
LeonGao91
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 2020-10-10 22:08:36 +05:30
bshashikant
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. (#2355) 2020-10-07 10:34:32 +05:30
Fei Hui
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356). Contributed by Hui Fei. 2020-10-05 22:11:24 +05:30
Siyao Meng
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir (#2352) 2020-10-05 05:02:00 -07:00
Takanobu Asanuma
d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Ye Ni
00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332) 2020-09-25 09:47:54 -07:00
huangtianhua
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
bshashikant
368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326) 2020-09-24 14:00:41 +05:30
Ayush Saxena
2377df38ad HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:18:10 +05:30
bshashikant
425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296) 2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-16 22:43:00 -07:00
Takanobu Asanuma
5c5b2ed7c7 HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui. 2020-09-16 01:36:46 +09:00