big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,452 Commits 2 Branches 2 Tags 264 MiB
fe0541b58d
Commit Graph

8902 Commits

Author SHA1 Message Date
GuoPhilipse
fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Steve Vaughan
f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
Viraj Jasani
17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
huhaiyang
d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Steve Vaughan
08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao
3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao
7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb
eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
Viraj Jasani
f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
Viraj Jasani
bce388fd3f
HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 09:11:02 +08:00
Masatake Iwasaki
a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
huhaiyang
88c8ac750d
HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-31 17:46:38 +09:00
Szilard Nemeth
b677d40ab5 HADOOP-18602. Remove netty3 dependency 2023-01-27 16:32:50 +01:00
Viraj Jasani
04f3573f6a
HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-18 13:13:41 -08:00
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
Viraj Jasani
1263e024b9
HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 14:26:52 -08:00
huhaiyang
e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-01-11 09:12:26 +08:00
Chengbing Liu
4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-01-10 10:03:25 -08:00
huhaiyang
f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
Yubi Lee
4511c360b9
HDFS-16883. Duplicate field name in hdfs-default.xml (#5271). Contributed by YUBI LEE. 2023-01-05 04:58:43 +05:30
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
Daniel-009497
7ff326129d
HDFS-16871 DiskBalancer process may throw IllegalArgumentException when the target DataNode has capital letter in hostname (#5240) 2022-12-21 07:12:02 +08:00
Happy-shi
c5b42d59d2
HDFS-16866. Fix a typo in Dispatcher (#5202) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-16 11:07:41 +08:00
curie71
fdcbc8b072
HDFS-16868. Fix audit log duplicate issue when an ACE occurs in FSNamesystem. (#5206). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-12-13 12:24:51 +08:00
Anurag P
e76616f690
HDFS-16860 Upgrade moment.min.js to 2.29.4 (#5194) 2022-12-09 11:18:44 +05:30
dingshun3016
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. (#5180) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-09 08:10:04 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. (#5142) 2022-11-29 00:49:10 +00:00
ZanderXu
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md (#4922) 2022-11-25 09:26:28 +08:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
ZanderXu
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap (#5125) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:07:27 +08:00
huhaiyang
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB (#5129) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:00:27 +08:00
huhaiyang
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate (#5063) 2022-11-23 09:50:57 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
ZanderXu
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. (#4945). Contributed by ZanderXu. 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-11-13 22:01:35 +08:00
Takanobu Asanuma
660530205e
HDFS-16833. NameNode should log internal EC blocks instead of the EC block group when it receives block reports. (#5106) 
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-07 10:49:48 +09:00
huhaiyang
e9319e696c
HDFS-16811. Support DecommissionBackoffMonitor Parameters reconfigurablereconfigurable (#5068) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-04 14:18:59 +08:00
jianghuazhu
88f7f5bc01
HDFS-16802.Print options when accessing ClientProtocol#rename2(). (#5013) 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
 2022-10-28 16:31:20 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:03:15 +08:00
huhaiyang
d26c35b228
HDFS-16817. Remove useless DataNode lock related configuration (#5072) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: ZanderXu <zanderxu@apache.org>
 2022-10-27 17:28:19 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-10-27 14:39:01 +09:00