big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,470 Commits 2 Branches 2 Tags 264 MiB
52ec65fd10
Commit Graph

9454 Commits

Author SHA1 Message Date
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin." 
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc.
 2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:09:37 +05:30
litao
2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568). Contributed by tomscut. 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:21:30 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
Gautham B A
a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform (#3563) 2021-10-21 10:24:33 -07:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
Xiangyi Zhu
d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 13:53:41 +09:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554) 
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
 2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Gautham B A
107fe227eb
HDFS-16267. Make hdfs_df tool cross platform (#3542) 2021-10-13 09:11:21 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497) 2021-10-13 17:21:03 +08:00
Gautham B A
a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532) 2021-10-11 18:25:28 -07:00
jianghuazhu
d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. (#3533) 2021-10-11 19:49:58 +08:00
langlaile1221
88d8c3d0c8
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
 2021-10-11 16:11:31 +08:00
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 15:32:49 +09:00
Gautham B A
4f3dfb7c1c
HDFS-16265. Refactor HDFS tool tests for better reuse (#3536) 2021-10-09 11:00:43 -07:00
Gautham B A
cc95fc4cc6
HDFS-16263. Add CMakeLists for hdfs_allowSnapshot (#3531) 2021-10-08 08:49:56 -07:00
Gautham B A
6bad3badf5
HDFS-16251. Make hdfs_cat tool cross platform (#3523) 2021-10-07 10:57:11 -07:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
Gautham B A
e12cd0c638
HDFS-16254. Cleanup protobuf on exit of hdfs_allowSnapshot (#3518) 2021-10-06 09:24:27 -07:00
S O'Donnell
a30231ff8e HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 2021-10-06 15:04:04 +01:00
Gautham B A
84f10fd78b
HDFS-16250. Refactor AllowSnapshotMock using GMock (#3513) 2021-10-05 09:16:42 -07:00
huhaiyang
68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511) 2021-10-04 23:52:26 +08:00
Ayush Saxena
5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-10-03 10:02:35 +05:30
Neil
bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494) 
Contributed by Max Xie
 2021-10-01 09:35:14 +02:00
jianghuazhu
1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. (#3483) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-10-01 10:30:41 +08:00
huhaiyang
1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471) 2021-09-28 13:09:18 +08:00
jianghuazhu
02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. (#3479) 2021-09-28 13:07:36 +08:00
Masatake Iwasaki
86e80752d1
HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. (#3492) 2021-09-28 13:35:35 +09:00
Chao Sun
1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" 
This reverts commit 59f01a548e.
 2021-09-25 21:32:43 -07:00
AngersZhuuuu
5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang
87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 2021-09-24 08:35:23 -07:00
litao
3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456) 2021-09-24 13:14:25 +08:00
Ayush Saxena
98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov. 2021-09-23 10:45:02 +05:30
Gautham B A
35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388) 2021-09-22 10:29:38 -07:00
Viraj Jasani
c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
LeonGao
b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Viraj Jasani
feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:11:20 +09:00
Tamas Domok
f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok 
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-20 14:29:50 +02:00
Thomas Leplus
f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Viraj Jasani
c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
jianghuazhu
cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-18 11:06:51 +05:30
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
LeonGao
deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
lfengnan
4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-16 14:58:02 +09:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
wangzhaohui
3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
 2021-09-15 09:56:51 -07:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
jianghuazhu
1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-14 14:24:36 +05:30
EungsopYoo
c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-09-14 13:18:15 +08:00
Ayush Saxena
d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404). Contributed by Ayush Saxena. 2021-09-13 22:50:39 +05:30
Ayush Saxena
17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-13 17:54:38 +08:00
bshashikant
356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) 2021-09-13 10:25:19 +05:30
jianghuazhu
edfde6eebc
HDFS-16197.Simplify getting NNStorage in FSNamesystem. (#3357) 2021-09-12 19:58:30 +08:00
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Symious
99cb2b6b7f
HDFS-16065. RBF: Add metrics to record Router's operations (#3100) 2021-09-09 18:45:23 +08:00
Ahmed Hussein
bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 2021-09-08 21:21:16 -07:00
Symious
c0890e6d04
HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 2021-09-09 09:57:27 +08:00
litao
5a306955a4
HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378) 2021-09-08 12:38:55 +08:00
jianghuazhu
4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368) 2021-09-06 18:06:12 +08:00
Renukaprasad C
dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244). Contributed by Renukaprasad C. 2021-09-05 17:07:59 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354). Contributed by tomscut. 2021-09-05 14:19:01 +08:00
Viraj Jasani
cc6b545365
HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-04 16:20:08 +05:30
Weison Wei
a610f6d9c3
HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367). Contributed by Weison Wei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-03 12:05:53 +05:30
langlaile1221
24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894) 2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2021-08-31 12:25:03 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Neil
9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315) 2021-08-24 11:15:47 +08:00
Gautham B A
b6d1971820
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311) 2021-08-20 10:04:55 -07:00
litao
07627ef19e
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-08-19 12:30:38 +08:00
litao
0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291) 2021-08-19 09:51:00 +08:00
Gautham B A
409878ef80
HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303) 2021-08-17 09:49:52 -07:00
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Gautham B A
77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Akira Ajisaka
4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
wangzhaohui
a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
 2021-07-29 11:19:17 +08:00
S O'Donnell
4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175) 
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
 2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185) 
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-26 12:45:01 +08:00