litao
dc751df63b
HDFS-16329. Fix log format for BlockManager ( #3670 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router ( #3635 )
2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )
2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers ( #3659 )
2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. ( #3320 )
2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode ( #3643 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP ( #3655 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable ( #3596 )
2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. ( #3629 )
2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics ( #3636 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-12 10:23:25 +09:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents ( #3637 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:22:32 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. ( #3622 )
2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics ( #3616 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability
2021-11-08 10:21:39 -08:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. ( #3605 )
2021-11-04 16:43:25 +09:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. ( #3602 )
2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log ( #3538 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file ( #3593 )
2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) ( #3598 )
2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. ( #3544 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-02 00:55:50 +09:00
jianghuazhu
0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). ( #3600 )
...
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600 ). Contributed by JiangHua Zhu.
2021-10-31 08:48:52 +05:30
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. ( #3374 )
2021-10-31 09:29:34 +09:00
litao
6aa664c624
HDFS-16279. Print detail datanode info when process first storage report ( #3564 )
2021-10-29 10:27:00 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout ( #3574 )
2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules ( #3566 )
2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin."
...
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc
.
2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:09:37 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy ( #3559 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a
HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker ( #3571 )
2021-10-21 13:41:46 -07:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery ( #3548 )
2021-10-18 11:55:55 +01:00
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt ( #3554 )
...
* HDFS-16274. Improve log for FSNamesystem#startFileInt
* fix failed unit test
2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE ( #3546 )
2021-10-13 18:14:03 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). ( #3497 )
2021-10-13 17:21:03 +08:00
jianghuazhu
d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. ( #3533 )
2021-10-11 19:49:58 +08:00
langlaile1221
88d8c3d0c8
HDFS-16241. Standby close reconstruction thread ( #3493 )
...
Co-authored-by: zhanghuazong <Zhz@20068867>
2021-10-11 16:11:31 +08:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules ( #3530 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-11 15:32:49 +09:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
S O'Donnell
a30231ff8e
HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell
2021-10-06 15:04:04 +01:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. ( #3422 ). Contributed by Ayush Saxena.
...
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. ( #3494 )
...
Contributed by Max Xie
2021-10-01 09:35:14 +02:00
jianghuazhu
1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. ( #3483 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-10-01 10:30:41 +08:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics ( #3471 )
2021-09-28 13:09:18 +08:00
jianghuazhu
02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. ( #3479 )
2021-09-28 13:07:36 +08:00
Chao Sun
1397cf2496
Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same"
...
This reverts commit 59f01a548e
.
2021-09-25 21:32:43 -07:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. ( #3468 )
2021-09-24 08:35:23 -07:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov.
2021-09-23 10:45:02 +05:30
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 ( #3427 )
2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink ( #3386 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:11:20 +09:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. ( #3455 ). Contributed by Thomas Leplus.
2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile ( #3429 )
2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. ( #3451 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-18 11:06:51 +05:30
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out ( #3425 )
2021-09-16 23:59:36 -07:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation ( #3366 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-09-16 10:00:02 +09:00
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount ( #3317 )
...
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
2021-09-15 09:56:51 -07:00
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. ( #2831 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown ( #3359 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-09-14 13:18:15 +08:00
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. ( #3404 ). Contributed by Ayush Saxena.
2021-09-13 22:50:39 +05:30
Ayush Saxena
17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. ( #3424 ). Contributed by Ayush Saxena.
2021-09-13 17:54:38 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing ( #3340 )
2021-09-13 10:25:19 +05:30
jianghuazhu
edfde6eebc
HDFS-16197.Simplify getting NNStorage in FSNamesystem. ( #3357 )
2021-09-12 19:58:30 +08:00
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router ( #3421 )
2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS ( #3346 ) Contributed by Leon Gao
...
* Router to support resolving monitored namenodes with DNS
* Style
* fix style and test failure
* Add test for NNHAServiceTarget const
* Resolve comments
* Fix test
* Comments and style
* Create a simple function to extract port
* Use LambdaTestUtils.intercept
* fix javadoc
* Trigger Build
2021-09-10 16:40:08 -07:00
Ahmed Hussein
bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query ( #3375 )
2021-09-08 21:21:16 -07:00
litao
5a306955a4
HDFS-16209. Add description for dfs.namenode.caching.enabled ( #3378 )
2021-09-08 12:38:55 +08:00
jianghuazhu
4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. ( #3368 )
2021-09-06 18:06:12 +08:00
Renukaprasad C
dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack ( #3244 ). Contributed by Renukaprasad C.
2021-09-05 17:07:59 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname ( #3354 ). Contributed by tomscut.
2021-09-05 14:19:01 +08:00
langlaile1221
24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue ( #2894 )
2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. ( #3353 )
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2021-08-31 12:25:03 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits ( #3235 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. ( #3284 )
...
* Add DNS resolution for QJM
* Add log
* Resolve comments
* checkstyle
* typo
2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile ( #3329 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-08-25 17:37:25 +09:00
Neil
9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. ( #3315 )
2021-08-24 11:15:47 +08:00
litao
07627ef19e
HDFS-16177. Bug fix for Util#receiveFile ( #3310 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-08-19 12:30:38 +08:00
litao
0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat ( #3291 )
2021-08-19 09:51:00 +08:00
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. ( #3295 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-08-17 16:47:10 +08:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus ( #3280 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map ( #3296 )
2021-08-16 13:40:05 +08:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely ( #3270 )
...
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-08-06 18:23:00 +08:00
wangzhaohui
a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager ( #3269 ). Contributed by wangzhaohui.
2021-08-06 09:27:05 +01:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… ( #3247 ) Contributed by Shuyan Zhang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-08-04 00:22:21 +08:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found ( #3243 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-30 08:29:56 -07:00
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. ( #2836 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
2021-07-29 11:19:17 +08:00
S O'Donnell
4eae284827
HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell
2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. ( #3175 )
...
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a
Co-authored-by: Zhihai Xu <zxu@apache.org>
2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. ( #3185 )
...
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically ( #3228 ). Contributed by Viraj Jasani.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. ( #3229 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web ( #3211 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-26 12:45:01 +08:00
Akira Ajisaka
2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. ( #3227 )
...
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"
This reverts commit b6d839a60c
.
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
2021-07-26 01:53:39 +09:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.
2021-07-17 21:57:00 +08:00
crossfire
4bb25c810b
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable ( #2414 )
2021-07-15 19:58:12 -07:00
Abhishek Das
1dd03cc4b5
HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das ( #2260 )
2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. ( #2639 )
...
* Rebase trunk
* Fix to use FQDN and update config name
* Fix javac
* Style and trigger build
* Trigger Build after force push
* Trigger Build
* Fix config names
2021-07-13 14:18:59 -07:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
S O'Donnell
b6c06c4b76
HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma
2021-07-09 10:03:57 +01:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. ( #3188 )
2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… ( #3140 )
2021-07-08 14:10:45 +08:00
JiaguodongF
9381071758
HDFS-16114. the balancer parameters print error ( #3179 )
...
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-07 14:13:05 +05:30
Ayush Saxena
c85a18327d
HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w.
2021-07-06 23:26:52 +05:30
Viraj Jasani
da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer ( #3169 )
2021-07-05 09:23:24 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout ( #3172 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-05 08:14:37 +09:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell ( #3168 )
2021-07-02 15:31:25 +08:00
zhangshuyan0
9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. ( #3159 )
...
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
2021-07-01 23:52:21 +08:00
litao
52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI ( #3160 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-01 18:44:10 +05:30
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing ( #3136 )
2021-06-30 22:06:56 -07:00
Viraj Jasani
d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts ( #3148 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack Chuang <stack@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-30 12:55:07 -07:00
Xiangyi Zhu
4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota ( #3156 )
...
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-06-30 00:12:08 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor ( #3146 )
2021-06-29 03:15:12 -07:00
Viraj Jasani
93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects ( #3150 )
2021-06-29 02:31:02 -07:00
litao
f78b6d47f9
HDFS-16085. Move the getPermissionChecker out of the read lock ( #3134 ). Contributed by tomscut.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-24 15:59:42 +05:30
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations ( #3117 )
2021-06-24 11:27:40 +09:00
Viraj Jasani
d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer ( #3127 )
2021-06-22 20:21:34 -07:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-06-21 10:25:12 +09:00
litao
bdc9c8809e
HDFS-16079. Improve the block state change log ( #3120 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:23:11 +05:30
Ahmed Hussein
c255feb5d7
HDFS-16061. DFTestUtil.waitReplication can produce false positives ( #3095 ). Contributed by Ahmed Hussein.
...
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:22:47 +05:30
litao
c491f81a30
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() ( #3119 ). Contributed by tomscut.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:20:52 +05:30
LeonGao
643dfd60e4
HDFS-15842. HDFS mover to emit metrics. ( #2738 )
2021-06-19 15:39:46 -07:00
hust_hhb
56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet ( #3065 )
2021-06-17 11:08:29 +08:00
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR ( #2998 )
...
Contributed by Viraj Jasani
2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode ( #3078 )
2021-06-14 10:48:22 -07:00
Ahmed Hussein
352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently ( #3097 )
2021-06-11 11:00:42 -05:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state ( #3084 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-06-11 17:30:55 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project ( #3073 )
2021-06-09 15:12:48 +09:00
Ahmed Hussein
a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based ( #3058 )
2021-06-08 11:09:31 -05:00
litao
c748fce17a
HDFS-16048. RBF: Print network topology on the router web ( #3062 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-08 16:14:06 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles ( #3036 )
...
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d
Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash ( #2927 )"
...
This reverts commit 041488e8f5
.
2021-06-01 08:38:13 +05:30
Ayush Saxena
bce14e746b
HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng.
2021-06-01 08:34:56 +05:30
Konstantin V Shvachko
8ce30f51f9
HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.
2021-05-27 18:47:20 -07:00
Konstantin V Shvachko
1abd03d68f
HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.
2021-05-26 14:59:50 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails ( #3052 )
2021-05-26 20:12:17 +09:00
Narges Shadab
3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap ( #3027 )
2021-05-26 14:43:41 +09:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS ( #3031 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 10:55:58 +09:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false ( #2747 )
2021-05-19 09:04:41 +08:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. ( #2860 ). Contributed by Ayush Saxena.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-05-16 00:45:11 +05:30
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException ( #2982 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-11 12:38:15 +09:00
Siyao Meng
8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all ( #2958 )
2021-05-10 12:41:07 -07:00
Ayush Saxena
359c0c1d1e
HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w.
2021-05-10 17:09:51 +05:30
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. ( #2980 )
...
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
2021-05-08 00:24:06 +09:00
zhuqi
76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. ( #2833 )
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-05-05 21:02:13 +05:30
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash ( #2927 )
...
Reviewed-by: Siyao Meng <siyao@apache.org>
2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). ( #2945 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type ( #2377 ) ( #2955 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories ( #2881 )
2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory ( #2849 ). Contributed by Stephen O'Donnell
2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean ( #2933 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-26 18:38:43 +09:00
Takanobu Asanuma
01bad0e92a
HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel.
2021-04-26 13:29:28 +09:00
Brahma Reddy Battula
2621d3f15b
HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650
Revert "HDFS-15624. fix the function of setting quota by storage type ( #2377 )"
...
This reverts commit 394b9f7a5c
.
Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
2021-04-26 11:27:15 +08:00
jianghuazhu
c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. ( #2913 ) Contributed by JiangHua Zhu.
2021-04-25 14:31:51 +08:00
Viraj Jasani
acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() ( #2923 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-21 19:45:49 +09:00
litao
85a3532849
HDFS-15970. Print network topology on the web ( #2896 )
2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong ( #2907 )
2021-04-17 16:49:09 +09:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. ( #2889 ) Contributed by Shuyan Zhang.
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-04-17 00:08:31 +08:00
Vivek Ratnavel Subramanian
c821008836
HDFS-15850. Superuser actions should be reported to external enforcers ( #2784 )
2021-04-15 14:45:51 -07:00
Viraj Jasani
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) ( #2874 )
2021-04-09 11:10:18 +09:00
Stephen O'Donnell
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell ( #2838 )
2021-04-08 11:59:02 +01:00
He Xiaoqiao
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.
2021-04-06 21:22:48 +08:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. ( #2862 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 13:39:10 +09:00
Viraj Jasani
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.( #2844 ). Contributed by Viraj Jasani
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 00:19:21 +05:30
zhuqi
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. ( #2829 ). Contributed by Qi Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-04-04 03:11:02 +05:30
litao
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker ( #2855 ). Contributed by tomscut.
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-02 10:37:04 -07:00
Brahma Reddy Battula
ed74479ea5
HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree.
2021-04-02 09:47:20 +05:30
litao
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog ( #2837 ). Contributed by tomscut.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-01 19:28:17 -07:00
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync ( #2770 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-02 10:33:40 +09:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management ( #2830 ). Contributed by Viraj Jasani
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-01 16:34:31 -07:00
Brahma Reddy Battula
0665ce9930
HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.
2021-04-01 09:19:39 +05:30
He Xiaoqiao
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.
2021-04-01 11:32:03 +08:00
Ayush Saxena
cd2501e54b
HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.
2021-03-28 16:13:59 +05:30
litao
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks ( #2748 )
...
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-03-27 17:57:09 +09:00
He Xiaoqiao
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.
2021-03-25 12:24:14 +08:00
touchida
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
2021-03-24 16:56:09 +08:00
Viraj Jasani
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result ( #2794 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 22:47:45 -07:00
Ayush Saxena
569e407f64
HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.
2021-03-24 02:10:02 +05:30
Jack Jiang
9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof ( #2805 )
2021-03-23 08:51:35 -07:00
Narges Shadab
d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal ( #2790 ). Contributed by Narges Shadab.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 17:06:58 +05:30
litao
343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2788 )
2021-03-22 20:29:17 +09:00
He Xiaoqiao
03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-22 18:37:00 +08:00
Viraj Jasani
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() ( #2785 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-19 20:37:44 -07:00
Narges Shadab
f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream ( #2736 ). Contributed by Narges Shadab.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-19 16:36:48 +08:00
Viraj Jasani
9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
2021-03-17 15:12:36 +09:00
Takanobu Asanuma
9ba60c33d8
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
2021-03-17 14:14:26 +09:00
touchida
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-17 12:00:05 +09:00
Hui Fei
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails ( #2779 )
2021-03-17 09:11:30 +08:00
sunlisheng
7025f39944
HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.
2021-03-15 11:34:13 +08:00
He Xiaoqiao
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.
2021-03-14 17:48:18 +08:00
lamberken
6921ec8b40
HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options ( #2751 ). Contributed by Xie Lei.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-13 14:22:11 +08:00
Xiaoyu Yao
7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. ( #2764 )
2021-03-12 11:27:53 -08:00
Takanobu Asanuma
ecd3335187
HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.
2021-03-12 15:42:15 +09:00
Hui Fei
6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate ( #2746 )
2021-03-10 14:11:29 +08:00
litao
6a922f9896
HDFS-15873. Add namenode address in logs for block report ( #2743 )
2021-03-08 11:50:51 -05:00
tomscut
9cb51bf106
HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. ( #2668 ) Contributed by tomscut.
2021-03-06 13:40:44 -08:00
litao
cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min ( #2739 )
...
Co-authored-by: tomscut <litao@bigo.sg>
2021-03-03 22:34:02 +09:00
zhuqi
1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. ( #2721 ). Contributed by Qi Zhu
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-02 13:16:11 +08:00
Konstantin V Shvachko
9501c698f4
HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.
2021-03-01 17:17:23 -08:00
litao
32353eb38a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker ( #2718 )
...
Authored-by: tomscut <litao@bigo.sg>
2021-03-01 07:52:59 -08:00
S O'Donnell
c3b3b36dee
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
2021-03-01 11:36:41 +00:00
S O'Donnell
9c583f36d5
HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee
2021-02-24 10:29:36 +00:00
LeonGao
940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. ( #2704 )
2021-02-23 22:29:48 -08:00
Ayush Saxena
647d23b611
HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.
2021-02-23 19:08:38 +05:30
Takanobu Asanuma
a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. ( #2714 )
2021-02-23 15:12:06 +09:00
Wang Yuxuan
1e32ab7d2f
HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang
f8059d0bdf
Revert "first commit ( #2559 )"
...
This reverts commit 1b94569d89
.
2021-02-22 18:13:06 +08:00
Wang Yuxuan
1b94569d89
first commit ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 02:10:39 -08:00
jianghuazhu
98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. ( #2686 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
2021-02-21 23:22:32 -08:00
Ayush Saxena
ff84a57483
HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.
2021-02-20 19:48:00 +05:30
Takanobu Asanuma
25af8901a9
HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.
2021-02-19 14:44:58 +09:00
Hui Fei
2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable ( #2694 )
2021-02-19 09:07:22 +08:00
Ayush Saxena
e391844e8e
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.
2021-02-18 01:39:04 +05:30
Akira Ajisaka
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level ( #2696 )
2021-02-17 01:59:24 -05:00
Zehao Chen
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes ( #2690 ). Contributed by Zehao Chen.
2021-02-15 11:14:32 -06:00
litao
c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics ( #2676 )
2021-02-14 00:12:24 -05:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. ( #2587 )
...
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.
The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.
Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
LeonGao
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao ( #2679 )
2021-02-09 10:49:28 +00:00
bshashikant
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. ( #2677 )
2021-02-09 10:27:51 +05:30
LeonGao
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. ( #2625 )
2021-02-08 11:01:21 -08:00
bshashikant
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit ( #2682 )
2021-02-06 08:26:12 -08:00
Jim Brennan
c4918fb298
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 21:20:30 +00:00
fermi
7ef2875c89
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
2021-02-05 11:01:49 +08:00
maobaolong
b99dba38b9
HDFS-15819. Fix import style #2681
2021-02-05 10:11:27 +08:00
Hui Fei
15a1f7adfc
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
2021-02-04 09:19:51 +08:00
Kihwal Lee
182623d2bc
HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.
2021-02-03 08:55:50 -06:00
Hui Fei
66ecee333e
HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang
2021-02-03 20:00:45 +08:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type ( #2377 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
crossfire
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka ( #2657 )
2021-02-02 09:02:09 +00:00
He Xiaoqiao
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-02 13:50:54 +08:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
He Xiaoqiao
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-01 23:28:29 +08:00
Narges Shadab
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
2021-02-01 01:56:59 -08:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer ( #2645 )
2021-02-01 13:42:44 +09:00
Jim Brennan
f565476939
[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.
2021-01-28 18:17:15 +00:00
Jim Brennan
6184ab07f6
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
2021-01-28 16:47:52 +00:00
sunlisheng
f8769e0f4b
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
2021-01-28 10:10:39 +08:00
Akira Ajisaka
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
2021-01-21 10:07:46 +09:00
He Xiaoqiao
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-16 14:06:56 +08:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount ( #2583 ). Contributed by Leon Gao.
2021-01-15 16:28:11 -08:00
Masatake Iwasaki
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-14 16:19:35 +09:00
touchida
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-13 11:23:07 +09:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. ( #2507 )
2021-01-11 11:59:32 -08:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
2021-01-08 13:10:09 -06:00
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics ( #2578 )
...
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
2021-01-04 09:43:58 -08:00
Masatake Iwasaki
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. ( #2501 )
2021-01-04 14:00:57 +09:00
Ayush Saxena
1cd96e8dd8
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.
2021-01-01 11:37:52 +05:30
maobaolong
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable ( #2572 )
2020-12-28 14:18:18 +08:00