Mukul Kumar Singh
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. ( #3694 )
2021-11-23 13:36:14 +00:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web ( #3682 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode ( #3654 )
2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md ( #3681 )
2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )" ( #3697 )
2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description ( #3678 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer ( #3663 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc ( #3669 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer ( #3672 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager ( #3670 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router ( #3635 )
2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )
2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers ( #3659 )
2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. ( #3320 )
2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode ( #3643 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster ( #3638 )
2021-11-16 09:47:25 +08:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms ( #3648 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-11-15 22:57:24 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP ( #3655 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable ( #3596 )
2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. ( #3629 )
2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics ( #3636 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-12 10:23:25 +09:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents ( #3637 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:22:32 +09:00
litao
ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException ( #3615 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-10 19:16:45 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. ( #3622 )
2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics ( #3616 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability
2021-11-08 10:21:39 -08:00
Gautham B A
0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp ( #3625 )
2021-11-08 08:58:07 -08:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice ( #3621 )
2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics ( #3553 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-08 09:30:53 +09:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice ( #3613 )
2021-11-05 11:03:07 +08:00
Gautham B A
1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp ( #3617 )
2021-11-04 09:19:11 -07:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. ( #3605 )
2021-11-04 16:43:25 +09:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. ( #3602 )
2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log ( #3538 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file ( #3593 )
2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) ( #3598 )
2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. ( #3544 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-02 00:55:50 +09:00
jianghuazhu
0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). ( #3600 )
...
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600 ). Contributed by JiangHua Zhu.
2021-10-31 08:48:52 +05:30
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. ( #3374 )
2021-10-31 09:29:34 +09:00
litao
6aa664c624
HDFS-16279. Print detail datanode info when process first storage report ( #3564 )
2021-10-29 10:27:00 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout ( #3574 )
2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules ( #3566 )
2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin."
...
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc
2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command ( #3576 ). Contributed by daimin.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 21:09:37 +05:30
litao
2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale ( #3568 ). Contributed by tomscut.
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 08:21:30 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy ( #3559 ). Contributed by guo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a
HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker ( #3571 )
2021-10-21 13:41:46 -07:00
Gautham B A
a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform ( #3563 )
2021-10-21 10:24:33 -07:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery ( #3548 )
2021-10-18 11:55:55 +01:00
Xiangyi Zhu
d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md ( #3557 )
...
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 13:53:41 +09:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang.
2021-10-16 14:35:18 +05:30
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt ( #3554 )
...
* HDFS-16274. Improve log for FSNamesystem#startFileInt
* fix failed unit test
2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE ( #3546 )
2021-10-13 18:14:03 -07:00
Gautham B A
107fe227eb
HDFS-16267. Make hdfs_df tool cross platform ( #3542 )
2021-10-13 09:11:21 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). ( #3497 )
2021-10-13 17:21:03 +08:00
Gautham B A
a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform ( #3532 )
2021-10-11 18:25:28 -07:00
jianghuazhu
d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. ( #3533 )
2021-10-11 19:49:58 +08:00
langlaile1221
88d8c3d0c8
HDFS-16241. Standby close reconstruction thread ( #3493 )
...
Co-authored-by: zhanghuazong <Zhz@20068867>
2021-10-11 16:11:31 +08:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules ( #3530 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-11 15:32:49 +09:00
Gautham B A
4f3dfb7c1c
HDFS-16265. Refactor HDFS tool tests for better reuse ( #3536 )
2021-10-09 11:00:43 -07:00
Gautham B A
cc95fc4cc6
HDFS-16263. Add CMakeLists for hdfs_allowSnapshot ( #3531 )
2021-10-08 08:49:56 -07:00
Gautham B A
6bad3badf5
HDFS-16251. Make hdfs_cat tool cross platform ( #3523 )
2021-10-07 10:57:11 -07:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
Gautham B A
e12cd0c638
HDFS-16254. Cleanup protobuf on exit of hdfs_allowSnapshot ( #3518 )
2021-10-06 09:24:27 -07:00
S O'Donnell
a30231ff8e
HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell
2021-10-06 15:04:04 +01:00
Gautham B A
84f10fd78b
HDFS-16250. Refactor AllowSnapshotMock using GMock ( #3513 )
2021-10-05 09:16:42 -07:00
huhaiyang
68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit ( #3511 )
2021-10-04 23:52:26 +08:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. ( #3422 ). Contributed by Ayush Saxena.
...
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. ( #3494 )
...
Contributed by Max Xie
2021-10-01 09:35:14 +02:00
jianghuazhu
1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. ( #3483 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-10-01 10:30:41 +08:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics ( #3471 )
2021-09-28 13:09:18 +08:00
jianghuazhu
02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. ( #3479 )
2021-09-28 13:07:36 +08:00
Masatake Iwasaki
86e80752d1
HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. ( #3492 )
2021-09-28 13:35:35 +09:00
Chao Sun
1397cf2496
Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same"
...
This reverts commit 59f01a548e
2021-09-25 21:32:43 -07:00
AngersZhuuuu
5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method ( #3472 )
2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. ( #3468 )
2021-09-24 08:35:23 -07:00
litao
3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader ( #3456 )
2021-09-24 13:14:25 +08:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov.
2021-09-23 10:45:02 +05:30
Gautham B A
35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform ( #3388 )
2021-09-22 10:29:38 -07:00
Viraj Jasani
c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename ( #3452 )
2021-09-22 15:34:51 +09:00
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 ( #3427 )
2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink ( #3386 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-21 10:11:20 +09:00
Tamas Domok
f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok
...
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.
The signature secret file was not used in HttpFs.
- if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
- if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly
!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.
The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-20 14:29:50 +02:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. ( #3455 ). Contributed by Thomas Leplus.
2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile ( #3429 )
2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. ( #3451 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-18 11:06:51 +05:30
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources ( #3434 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-17 16:39:29 +09:00
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out ( #3425 )
2021-09-16 23:59:36 -07:00
lfengnan
4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal ( #3444 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-16 14:58:02 +09:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation ( #3366 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-09-16 10:00:02 +09:00
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount ( #3317 )
...
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
2021-09-15 09:56:51 -07:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils ( #3428 ). Contributed by tomscut.
2021-09-14 14:28:19 +05:30
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. ( #2831 ). Contributed by JiangHua Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown ( #3359 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-09-14 13:18:15 +08:00
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. ( #3404 ). Contributed by Ayush Saxena.
2021-09-13 22:50:39 +05:30
Ayush Saxena
17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. ( #3424 ). Contributed by Ayush Saxena.
2021-09-13 17:54:38 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing ( #3340 )
2021-09-13 10:25:19 +05:30
