big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,173 Commits 2 Branches 2 Tags 264 MiB
94b3c6dd90
Commit Graph

9305 Commits

Author SHA1 Message Date
rdingankar
94b3c6dd90 HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 15:49:26 -08:00
Owen O'Malley
c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira
5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu
9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-24 13:31:53 -08:00
hchaverr
eab7215354
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 14:02:13 -08:00
Viraj Jasani
e47ff5f3c0
HDFS-16925. Namenode audit log to only include IP address of client (#5413) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:33:54 +08:00
Steve Vaughan
d437571fe2
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:47:00 +00:00
Viraj Jasani
5099077f4d
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) (#5358) 
* ADDENDUM Remove unused variables from testDataNodeMXBeanLastHeartbeats (#5373)

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-12 11:53:59 +05:30
Ayush Saxena
b6d732b96b
HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2023-02-12 01:57:34 +05:30
Masatake Iwasaki
d85c12ee69
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2023-02-12 01:50:39 +05:30
Ahmed Hussein
c17734b747
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2023-02-12 01:50:26 +05:30
huhaiyang
de08baded6
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:29:47 +00:00
Steve Vaughan
b6680907f9
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-09 13:29:00 +00:00
Tao Li
10b42e9306 HDFS-16902. Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 10:02:45 +08:00
huhaiyang
80b42625cf HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 88c8ac750d)
 2023-01-31 17:52:29 +09:00
Viraj Jasani
63443be5f4 HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 04f3573f6a)
 2023-01-18 21:14:17 +00:00
Viraj Jasani
c5cf845d78 HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 1263e024b9)
 2023-01-12 22:27:35 +00:00
Simbarashe Dzinamarira
dde22d522c HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 11:47:57 -08:00
Chengbing Liu
af96e0f5b3 HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 4cf304de45)
 2023-01-10 10:04:05 -08:00
ZanderXu
a49378cb93 HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>

(cherry picked from commit 8d221255f2)
 2023-01-10 10:03:45 -08:00
Xing Lin
d43fa95043 HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down (#5160) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit f7bdf6c667)
 2022-12-16 08:47:33 -08:00
dingshun3016
ebd2407d48 HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 
(cherry picked from commit 02afb9ebe1)
 2022-12-05 16:39:49 +09:00
Ashutosh Gupta
fa2a0a603a HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
(cherry picked from commit b7edc6c60c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java
 2022-12-01 15:00:32 +09:00
Kidd5368
8c7f2ddc10
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
(cherry picked from commit 72749a4ff8)
 2022-11-29 17:45:12 -08:00
Owen O'Malley
6b23e70539 HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 14:19:19 -08:00
Simbarashe Dzinamarira
9d37ee082c HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-28 16:54:25 -08:00
zhengchenyu
4addf31ef4 HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>

(cherry picked from commit dc2fba45fe)
 2022-11-21 08:33:16 -08:00
Owen O'Malley
8ff54dac58 HDFS-16844: RBF: Adds resilancy when StateStore gets exceptions. (#5138) 
Allows the StateStore to stay up when there are errors reading the data.
 2022-11-18 09:29:08 -08:00
Lei Yang
b68520d2a5 HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 15:10:41 -08:00
huhaiyang
033ceca090
HDFS-16811. Support DecommissionBackoffMonitor parameters reconfigurable (#5122) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-10 13:37:09 +08:00
Melissa You
02aedd7811
Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 (#5112) 
* HDFS-15383. RBF: Add support for router delegation token without watch (#2047)

Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.

* HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: lfengnan <lfengnan@uber.com>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
 2022-11-07 13:29:50 -08:00
Melissa You
853ffb545a
HADOOP-18515. Backport HADOOP-17612 to branch-3.3(Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0) (#5097) 
* HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
 2022-11-05 09:28:24 -07:00
M1eyu2018
cbac2c4875 HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:11:51 +08:00
Takanobu Asanuma
53143409a8 HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
(cherry picked from commit 545a556883)
 2022-10-27 14:40:07 +09:00
Viraj Jasani
36a0e818ec HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani

(cherry picked from commit c1bf3cb0da)
 2022-10-24 15:16:38 +09:00
Takanobu Asanuma
198bc444de
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) (#5059) 
(cherry picked from commit 9376b65989)

Co-authored-by: RuinanGu <57645247+RuinanGu@users.noreply.github.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-22 13:14:04 +09:00
SevenAddSix
237814a9b3 HDFS-16480. Fix typo: indicies -> indices (#4020) 
(cherry picked from commit 5eab9719cb)
 2022-10-21 17:32:58 +09:00
Hui Fei
0c2234fd8e HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang 
(cherry picked from commit 66ecee333e)
 2022-10-21 17:31:30 +09:00
Steve Loughran
75b04010a2
HDFS-16795. Use secure XML parsers (#4979) 
Move construction of XML parsers in HDFS
modules to using the locked-down parser factory
of HADOOP-18469.

Contributed by P J Fanning
 2022-10-20 17:48:58 +01:00
Xing Lin
760144f135
HDFS-16628. RBF: Correct target directory when move to trash for kerberos login user. (#4974) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-11 16:14:39 +09:00
Ashutosh Gupta
6847ec0647
HADOOP-11245. Update NFS gateway to use Netty4 (#2832) (#4997) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-10-11 05:27:43 +08:00
Xing Lin
7d7f7a9e9b
HDFS-16024. RBF: Rename data to the Trash should be based on src location (#4962) 
(cherry picked from commit e18d806212)

Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:33:48 +09:00
Ashutosh Gupta
dea018ef23
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source (#4886) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d9f435f6ac)
 2022-09-27 15:44:58 +09:00
Kidd5368
ceec19e61a HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901) 
(cherry picked from commit 9a29075f91)
 2022-09-23 12:29:39 +09:00
Steve Vaughan
357c83db94
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724) (#4794) 2022-09-13 10:50:23 -07:00
Steve Vaughan
3a6c8ff8bb
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833) 
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr
 2022-09-01 14:01:26 +01:00
Steve Vaughan
833fc64558
HDFS-16684. Exclude the current JournalNode (#4786) 
The JournalNodeSyncer will include the local instance in syncing when using a bind host (e.g. 0.0.0.0).  There is a mechanism that is supposed to exclude the local instance, but it doesn't recognize the meta-address as a local address.

Running with bind addresses set to 0.0.0.0, the JournalNodeSyncer will log attempts to sync with itself as part of the normal syncing rotation.  For an HA configuration running 3 JournalNodes, the "other" list used by the JournalNodeSyncer will include 3 proxies.

Exclude bound local addresses, including the use of a wildcard address in the bound host configurations, while still allowing multiple instances on the same host.

Allow sync attempts with unresolved addresses, so that sync attempts can drive resolution as servers become available.

Backport.
Signed-off-by: stack <stack@apache.org>
 2022-08-28 11:15:04 -07:00
zhengchenyu
3edddaf9fc HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 231a4468cd)
 2022-08-25 10:41:04 -07:00
Simba Dzinamarira
e28dc524f6
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:27:21 -07:00
Steve Vaughan
4138661010
HDFS-16625. Check assumption about PMDK availability (#4788) 
Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
 2022-08-23 19:35:59 +09:00