big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,118 Commits 2 Branches 2 Tags 264 MiB
7b239a80fe
Commit Graph

9718 Commits

Author SHA1 Message Date
PHILO-HE
1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761) 2021-12-08 17:52:40 +05:30
GuoPhilipse
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode (#3661) 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 21:21:48 +09:00
lzx404243
7ba91c1235
HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 20:44:18 +09:00
GuoPhilipse
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-08 16:34:54 +05:30
Masatake Iwasaki
e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740) 2021-12-07 21:39:04 +09:00
GuoPhilipse
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem (#3713). Contributed by guophilipse. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-07 14:19:27 +05:30
GuoPhilipse
1509ea5de2
HDFS-16338. Correct fsimage error configuration message (#3684). Contributed by guophilipse. 
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-06 14:00:50 +05:30
Takanobu Asanuma
e8e69de106 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 2021-12-06 10:44:36 +09:00
litao
c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo (#3747). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 22:54:44 +05:30
GuoPhilipse
2f55a6cced
HDFS-16357. Fix log format in DFSUtilClient (#3729). Contributed by guophilipse. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 18:35:05 +05:30
Ayush Saxena
cab7086fbc
HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745). Contributed by Ayush Saxena. 
Reviewed-by: litao <tomleescut@gmail.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-12-04 10:54:29 +05:30
huhaiyang
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-04 02:19:36 +09:00
bitterfox
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-03 23:30:13 +09:00
wangzhaohui
0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 (#3742) 2021-12-03 16:51:04 +05:30
litao
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-03 14:12:05 +09:00
Viraj Jasani
0c62a514f7
HDFS-16358. HttpFS implementation for getSnapshotDiffReportListing (#3730) 2021-12-03 13:25:32 +09:00
litao
196935a8d2
HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731). Contributed by tomscut. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-02 19:10:35 +05:30
litao
1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand (#3732). Contributed by tomscut. 2021-12-01 12:48:21 +05:30
litao
829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-29 17:48:10 +09:00
litao
86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-26 19:52:14 +09:00
Takanobu Asanuma
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-25 14:05:04 +09:00
Viraj Jasani
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 2021-11-25 10:23:04 +08:00
huhaiyang
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Viraj Jasani
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 22:54:11 +09:00
Mukul Kumar Singh
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 2021-11-23 13:36:14 +00:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md (#3681) 2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629) 2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 10:23:25 +09:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:22:32 +09:00
litao
ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:16:45 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability 2021-11-08 10:21:39 -08:00
Gautham B A
0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625) 2021-11-08 08:58:07 -08:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Gautham B A
1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617) 2021-11-04 09:19:11 -07:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 2021-11-04 16:43:25 +09:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602) 2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-02 00:55:50 +09:00
jianghuazhu
0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600) 
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600). Contributed by JiangHua Zhu.
 2021-10-31 08:48:52 +05:30
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
litao
6aa664c624
HDFS-16279. Print detail datanode info when process first storage report (#3564) 2021-10-29 10:27:00 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin." 
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc.
 2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:09:37 +05:30
litao
2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568). Contributed by tomscut. 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:21:30 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
Gautham B A
a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform (#3563) 2021-10-21 10:24:33 -07:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
Xiangyi Zhu
d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 13:53:41 +09:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554) 
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
 2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Gautham B A
107fe227eb
HDFS-16267. Make hdfs_df tool cross platform (#3542) 2021-10-13 09:11:21 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497) 2021-10-13 17:21:03 +08:00
Gautham B A
a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532) 2021-10-11 18:25:28 -07:00
jianghuazhu
d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. (#3533) 2021-10-11 19:49:58 +08:00
langlaile1221
88d8c3d0c8
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
 2021-10-11 16:11:31 +08:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 15:32:49 +09:00
Gautham B A
4f3dfb7c1c
HDFS-16265. Refactor HDFS tool tests for better reuse (#3536) 2021-10-09 11:00:43 -07:00
Gautham B A
cc95fc4cc6
HDFS-16263. Add CMakeLists for hdfs_allowSnapshot (#3531) 2021-10-08 08:49:56 -07:00
Gautham B A
6bad3badf5
HDFS-16251. Make hdfs_cat tool cross platform (#3523) 2021-10-07 10:57:11 -07:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
Gautham B A
e12cd0c638
HDFS-16254. Cleanup protobuf on exit of hdfs_allowSnapshot (#3518) 2021-10-06 09:24:27 -07:00
S O'Donnell
a30231ff8e HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 2021-10-06 15:04:04 +01:00
Gautham B A
84f10fd78b
HDFS-16250. Refactor AllowSnapshotMock using GMock (#3513) 2021-10-05 09:16:42 -07:00
huhaiyang
68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511) 2021-10-04 23:52:26 +08:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494) 
Contributed by Max Xie
 2021-10-01 09:35:14 +02:00
jianghuazhu
1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. (#3483) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-10-01 10:30:41 +08:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471) 2021-09-28 13:09:18 +08:00
jianghuazhu
02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. (#3479) 2021-09-28 13:07:36 +08:00
Masatake Iwasaki
86e80752d1
HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. (#3492) 2021-09-28 13:35:35 +09:00
Chao Sun
1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" 
This reverts commit 59f01a548e.
 2021-09-25 21:32:43 -07:00
AngersZhuuuu
5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 2021-09-24 08:35:23 -07:00
litao
3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456) 2021-09-24 13:14:25 +08:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov. 2021-09-23 10:45:02 +05:30
Gautham B A
35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388) 2021-09-22 10:29:38 -07:00
Viraj Jasani
c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:11:20 +09:00
Tamas Domok
f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok 
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-20 14:29:50 +02:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-18 11:06:51 +05:30
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
lfengnan
4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-16 14:58:02 +09:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
 2021-09-15 09:56:51 -07:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-09-14 13:18:15 +08:00
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404). Contributed by Ayush Saxena. 2021-09-13 22:50:39 +05:30
Ayush Saxena
17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-13 17:54:38 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) 2021-09-13 10:25:19 +05:30
jianghuazhu
edfde6eebc
HDFS-16197.Simplify getting NNStorage in FSNamesystem. (#3357) 2021-09-12 19:58:30 +08:00
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Symious
99cb2b6b7f
HDFS-16065. RBF: Add metrics to record Router's operations (#3100) 2021-09-09 18:45:23 +08:00
Ahmed Hussein
bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 2021-09-08 21:21:16 -07:00
Symious
c0890e6d04
HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 2021-09-09 09:57:27 +08:00
litao
5a306955a4
HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378) 2021-09-08 12:38:55 +08:00
jianghuazhu
4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368) 2021-09-06 18:06:12 +08:00
Renukaprasad C
dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244). Contributed by Renukaprasad C. 2021-09-05 17:07:59 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354). Contributed by tomscut. 2021-09-05 14:19:01 +08:00
Viraj Jasani
cc6b545365
HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-04 16:20:08 +05:30
Weison Wei
a610f6d9c3
HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367). Contributed by Weison Wei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-03 12:05:53 +05:30
langlaile1221
24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894) 2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2021-08-31 12:25:03 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Neil
9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315) 2021-08-24 11:15:47 +08:00
Gautham B A
b6d1971820
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311) 2021-08-20 10:04:55 -07:00
litao
07627ef19e
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-08-19 12:30:38 +08:00
litao
0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291) 2021-08-19 09:51:00 +08:00
Gautham B A
409878ef80
HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303) 2021-08-17 09:49:52 -07:00
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Gautham B A
77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Akira Ajisaka
4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
wangzhaohui
a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
 2021-07-29 11:19:17 +08:00
S O'Donnell
4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175) 
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
 2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185) 
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-26 12:45:01 +08:00
Akira Ajisaka
2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2021-07-26 01:53:39 +09:00
Eric Yin
de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00
Kihwal Lee
47002719f2 HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 2021-07-16 14:22:39 -05:00
crossfire
4bb25c810b
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 2021-07-15 19:58:12 -07:00
Abhishek Das
1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
S O'Donnell
b6c06c4b76 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 2021-07-09 10:03:57 +01:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
JiaguodongF
9381071758
HDFS-16114. the balancer parameters print error (#3179) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-07 14:13:05 +05:30
Ayush Saxena
c85a18327d HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w. 2021-07-06 23:26:52 +05:30
litao
a447cfd78b
HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-07-05 19:31:22 -07:00
Viraj Jasani
da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 2021-07-05 09:23:24 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-05 08:14:37 +09:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
zhangshuyan0
9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. (#3159) 
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
 2021-07-01 23:52:21 +08:00
litao
52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-01 18:44:10 +05:30
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
Viraj Jasani
d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:55:07 -07:00
Xiangyi Zhu
4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-06-30 00:12:08 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
Viraj Jasani
93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 2021-06-29 02:31:02 -07:00
He Xiaoqiao
748570b73c
Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." 
This reverts commit 0d07837712.
 2021-06-25 22:38:34 +08:00
He Xiaoqiao
0d07837712
HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun. 2021-06-25 01:09:42 +08:00
litao
f78b6d47f9
HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 15:59:42 +05:30
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Gautham B A
ef5dbc7283
HADOOP-17766. CI for Debian 10 (#3129) 2021-06-23 10:02:33 -07:00
Viraj Jasani
d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 2021-06-22 20:21:34 -07:00
He Xiaoqiao
10b79a26fe
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-22 00:32:55 +08:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
litao
bdc9c8809e
HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:11 +05:30
Viraj Jasani
d9eccb1eb3
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:19:05 +05:30
Ahmed Hussein
c255feb5d7
HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:22:47 +05:30
litao
c491f81a30
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:20:52 +05:30
LeonGao
643dfd60e4
HDFS-15842. HDFS mover to emit metrics. (#2738) 2021-06-19 15:39:46 -07:00
Wei-Chiu Chuang
1ad674e548
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
 2021-06-16 20:37:56 -07:00
hust_hhb
56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065) 2021-06-17 11:08:29 +08:00
Ayush Saxena
2b304ad645 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:53:12 +05:30
Wei-Chiu Chuang
a77bf7cf07 Make upstream aware of 3.3.1 release 2021-06-15 15:43:36 +00:00
Takanobu Asanuma
839fcf7682
HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 2021-06-15 13:20:09 +05:30
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani
 2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
Ahmed Hussein
352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097) 2021-06-11 11:00:42 -05:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-11 17:30:55 +09:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
Ayush Saxena
a1a3184171
HDFS-15916. Addendum. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#3056) 2021-06-08 19:34:45 -07:00
Ahmed Hussein
a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058) 2021-06-08 11:09:31 -05:00
litao
c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena
bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Jinglun
e90c41af34 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 19:28:40 +08:00
Jinglun
2b11e160e2 Revert "HDFS-15973." because the user.name and user.email message is not correct. 
This reverts commit b7f69467c1.
 2021-05-30 19:27:10 +08:00
李经纶
b7f69467c1 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 18:50:03 +08:00
Konstantin V Shvachko
8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko
1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Gautham B A
bcaeb1ac8c
Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898)"" (#3044) 2021-05-26 14:12:53 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Narges Shadab
3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
zhuxiangyi
e18d806212
HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009) 2021-05-26 13:41:49 +08:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
lfengnan
43bf009112
HDFS-15757 RBF: Improving Router Connection Management (#2651) 2021-05-19 10:53:42 -07:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Siyao Meng
8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958) 2021-05-10 12:41:07 -07:00
Ayush Saxena
359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
Jinglun
1b6994287a HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819). Contributed by zhuobin zheng. 
Reviewed-by: Jinglun <jinglun@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-08 19:50:17 +08:00
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
zhuqi
76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Akira Ajisaka
7c561e2a02
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970) 2021-05-04 00:25:29 +09:00
lfengnan
6e525ab81c
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Karthik Palanisamy
bd3da73a0f
HDFS-15865. Interrupt DataStreamer thread if no ack (#2728) 2021-05-01 23:35:31 +05:30
lfengnan
552e9dcc6c
HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954). Contributed by Fengnan Li. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang
f41a368c14
HADOOP-11245. Update NFS gateway to use Netty4 (#2832) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2021-04-28 23:43:11 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma
01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu
c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
zhuxiangyi
dff95c5eca
HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 10:19:11 +09:00
Viraj Jasani
acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao
85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
Eric Badger
aed13f0f42 Revert "HDFS-15971. Make mkstemp cross platform (#2898)" 
This reverts commit b088f46b68.
 2021-04-16 17:15:33 +00:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Akira Ajisaka
f0241ec216
HDFS-15977. Call explicit_bzero only if it is available. (#2914) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-04-16 13:26:45 +09:00
lfengnan
17be99f9f9
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903) 2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian
c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Masatake Iwasaki
2717203f85
HADOOP-17569. Building native code fails on Fedora 33. (#2886) 2021-04-15 21:33:18 +09:00
Ayush Saxena
7d6f0ca0bd Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." 
This reverts commit e565b05c80.

The removed method needs to be used in HDFS-15423.
 2021-04-13 23:28:42 +05:30
Gautham B A
b088f46b68
HDFS-15971. Make mkstemp cross platform (#2898) 2021-04-13 08:54:45 -07:00
Inigo Goiri
82462739f8
Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) 
This reverts commit cb3ed32fe0.
 2021-04-12 20:44:31 -07:00
lfengnan
cb3ed32fe0
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605) 2021-04-12 12:42:33 -07:00
Gautham B A
3148791da4
HDFS-15962. Make strcasecmp cross platform (#2883) 2021-04-09 09:01:33 -07:00
Viraj Jasani
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Gautham B A
bf66116407
HDFS-15955. Make explicit_bzero cross platform (#2875) 2021-04-08 09:44:47 -07:00
Ayush Saxena
c6539e3289
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-08 20:49:08 +05:30
Stephen O'Donnell
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
Gautham B A
42ddb5c6fe
HDFS-15909. Make fnmatch cross platform (#2792) 2021-04-06 09:23:21 -07:00
He Xiaoqiao
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Gautham B A
36014b8282
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-06 18:22:54 +09:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
Gautham B A
6e1df1c048
HDFS-15947. Replace deprecated protobuf APIs (#2856) 2021-04-05 09:52:48 -07:00
litao
9e2d5d6d05
HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-05 17:47:10 +09:00
Gautham B A
ca5e13fd74
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-05 11:42:05 +09:00
zhuqi
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Gautham B A
70536ba1f9
HDFS-15944. Prevent truncation by snprintf (#2853) 2021-04-02 09:37:33 -07:00
Brahma Reddy Battula
ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula
0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
He Xiaoqiao
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Borislav Iordanov
2c482fbacf HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:46:35 -07:00
stack
22961a615d Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" 
This reverts commit a2975d2153.
 2021-03-31 10:43:09 -07:00
stack
a2975d2153 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
 2021-03-31 10:40:20 -07:00
stack
5183aaeda2 Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693)" 
Revert to fix the summary message.

This reverts commit 9509bebf7f.
 2021-03-31 10:39:55 -07:00
Borislav Iordanov
9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes (#2693) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:07:42 -07:00
He Xiaoqiao
da4ceba4aa
HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C. 2021-03-31 13:00:13 +08:00
Gautham B A
7dfff496fa
HDFS-15929. Replace RAND_pseudo_bytes in util.cc (#2826) 2021-03-30 13:36:02 -07:00
Gautham B A
8668abf87e
HDFS-15927. Catch polymorphic type by reference (#2824) 2021-03-30 13:35:45 -07:00
Gautham B A
50b1e1c0c4
HDFS-15928. Replace RAND_pseudo_bytes in rpc_engine.cc (#2825) 2021-03-30 10:37:10 -07:00
Viraj Jasani
90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:22:54 +05:30