big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,205 Commits 2 Branches 2 Tags 264 MiB
0b56be3ca4
Commit Graph

9314 Commits

Author SHA1 Message Date
Xing Lin
7d7f7a9e9b
HDFS-16024. RBF: Rename data to the Trash should be based on src location (#4962) 
(cherry picked from commit e18d806212)

Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:33:48 +09:00
Ashutosh Gupta
dea018ef23
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source (#4886) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d9f435f6ac)
 2022-09-27 15:44:58 +09:00
Kidd5368
ceec19e61a HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901) 
(cherry picked from commit 9a29075f91)
 2022-09-23 12:29:39 +09:00
Steve Vaughan
357c83db94
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724) (#4794) 2022-09-13 10:50:23 -07:00
Steve Vaughan
3a6c8ff8bb
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833) 
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr
 2022-09-01 14:01:26 +01:00
Steve Vaughan
833fc64558
HDFS-16684. Exclude the current JournalNode (#4786) 
The JournalNodeSyncer will include the local instance in syncing when using a bind host (e.g. 0.0.0.0).  There is a mechanism that is supposed to exclude the local instance, but it doesn't recognize the meta-address as a local address.

Running with bind addresses set to 0.0.0.0, the JournalNodeSyncer will log attempts to sync with itself as part of the normal syncing rotation.  For an HA configuration running 3 JournalNodes, the "other" list used by the JournalNodeSyncer will include 3 proxies.

Exclude bound local addresses, including the use of a wildcard address in the bound host configurations, while still allowing multiple instances on the same host.

Allow sync attempts with unresolved addresses, so that sync attempts can drive resolution as servers become available.

Backport.
Signed-off-by: stack <stack@apache.org>
 2022-08-28 11:15:04 -07:00
zhengchenyu
3edddaf9fc HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 231a4468cd)
 2022-08-25 10:41:04 -07:00
Simba Dzinamarira
e28dc524f6
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:27:21 -07:00
Steve Vaughan
4138661010
HDFS-16625. Check assumption about PMDK availability (#4788) 
Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
 2022-08-23 19:35:59 +09:00
Steve Vaughan
a73efb2d55
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) (#4790) 2022-08-22 20:26:03 -07:00
jianghuazhu
2123859d60
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f176d080c)
 2022-08-21 07:31:01 +09:00
Steve Loughran
9c5228cf6b
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.

Change-Id: I98b0fed54da3b810c3f23fe5b12e673937916257
 2022-08-05 14:02:28 +01:00
xuzq
e024d1a3f8
HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-03 13:02:42 +05:30
Ayush Saxena
df4e59318f
HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun. 
Only Prod Changes: Test already cherry-picked as part of HDFS-16310 via (496657c63f)
 2022-07-23 23:48:02 +05:30
Masatake Iwasaki
ff13f9ee8b Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57dfd41385dbce5de90620c48a45abb263)
 2022-07-22 02:31:34 +00:00
Masatake Iwasaki
75c739c458 Revert "HADOOP-17196. Fix C/C++ standard warnings (#2208)" 
This reverts commit b4a105a209.
 2022-06-30 00:57:52 +00:00
Viraj Jasani
4ba463069b
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4485) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-23 17:30:01 +08:00
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482) 
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
 2022-06-22 13:09:50 +01:00
Viraj Jasani
7561dbd134
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:44:30 +05:30
KevinWikant
33ab84f2e2 HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6)
 2022-06-20 11:24:45 +09:00
jianghuazhu
18a5e843bc HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:20:34 +08:00
xuzq
ee3ee98ee5 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473)
 2022-06-10 19:02:47 +00:00
Ashutosh Gupta
bdef321d52 HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
 2022-06-09 22:42:04 +09:00
slfan1989
a2f8a9e5d8 HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:59:34 +08:00
Viraj Jasani
516a2a8e44 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:56:07 +08:00
Viraj Jasani
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) (#4405) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-07 06:41:16 +08:00
Samrat
7223a337f6 HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1d)
 2022-06-06 18:02:44 +09:00
Stephen O'Donnell
7d6b133af3 HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2022-06-01 20:54:56 +01:00
Ashutosh Gupta
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785)
 2022-05-26 15:28:27 -07:00
Stephen O'Donnell
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a)
 2022-05-26 10:14:50 -07:00
Wei-Chiu Chuang
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) (#4304) 
(cherry picked from commit cee8c62498)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java

(cherry picked from commit dd79aee635fdc61648e0c87bea1560dc35aee053)

Co-authored-by: caozhiqiang <lfxy@163.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-27 00:50:40 +08:00
Michael Stack
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4347) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 17:02:28 -07:00
jianghuazhu
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. (#4342). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-24 23:47:45 +08:00
Viraj Jasani
ab3a9cedc9 HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:47:18 +08:00
Ashutosh Gupta
57fe613299
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb910bd906)

 Conflicts:
	hadoop-project/pom.xml
 2022-05-21 03:17:15 +09:00
Steve Loughran
fe306ce57e HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:09 +01:00
Michael Stack
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) (#4246) 
Cherry-pick backport of 9ed8d60511
 2022-05-15 21:32:48 -07:00
Viraj Jasani
e16ae55833 HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 09:00:31 +08:00
daimin
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4296) 2022-05-11 07:56:53 +08:00
Ashutosh Gupta
c879230547
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9)
 2022-05-08 08:06:14 +09:00
Ashutosh Gupta
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5b)
 2022-05-08 07:39:17 +09:00
daimin
d711d200fe
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29401c8203)
 2022-05-06 10:35:55 -07:00
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Viraj Jasani
ee450bbbc6
HDFS-16528. Reconfigure slow peer enable for Namenode (#4251) 2022-05-02 08:03:02 +08:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
daimin
b62a460fd9 HDFS-16519. Add throttler to EC reconstruction (#4101) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit aebd55f788)
 2022-04-23 12:47:34 +09:00
litao
9132eeb4dd HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc480)
 2022-04-23 12:15:52 +09:00
S O'Donnell
5e137ac33e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 8ae033d1a3.
 2022-04-20 20:45:17 +01:00
qinyuren
2ff91232bc HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd17374)
 2022-04-20 15:07:38 +09:00
Stephen O'Donnell
8ae033d1a3 HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee0363)
 2022-04-19 11:12:36 +01:00
qinyuren
c913dc3072 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0)
 2022-04-19 13:38:58 +09:00
jianghuazhu
cfe2d8aa79
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e712)
 2022-04-19 11:13:40 +08:00
Ashutosh Gupta
1eb4f9ef04
HDFS-16536. TestOfflineImageViewer fails on branch-3.3 (#4182) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-18 16:05:42 +09:00
Quanlong Huang
9ae903dd1b
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
(cherry picked from commit 35d4c02bcc)
 2022-04-18 10:35:08 +08:00
daimin
0ef1a13f01
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 
(cherry picked from commit c65c383b7e)
 2022-04-14 16:16:36 +08:00
Takanobu Asanuma
52abc9f132 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959)
 2022-04-14 11:50:42 +09:00
Steve Loughran
52c6d77274
HDFS-14478. Add libhdfs APIs for openFile (#4166) 
Contributed by Sahil Takiar

Change-Id: I2f9e82a69010df7496704754515b031f2a907167
 2022-04-13 16:01:53 +01:00
qinyuren
07dface36a HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a1)
 2022-04-13 11:48:19 +09:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
singer-bin
26705bbc60
HDFS-16457. Make fs.getspaceused.classname reconfigurable (apache#4069) (#4156) 2022-04-11 14:59:34 +09:00
Takanobu Asanuma
30afe7ca20 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 
(cherry picked from commit 37650ced81)
 2022-04-08 18:39:26 +09:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00
Stephen O'Donnell
bd0dbf319a HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf)
 2022-04-06 11:54:58 +01:00
Viraj Jasani
b2eee14f2e HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7c20602b17)
 2022-04-06 18:55:09 +09:00
Viraj Jasani
a6fb77f7eb HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 
(cherry picked from commit 278568203b)
 2022-04-06 18:40:07 +09:00
wangzhaohui
0e621c890d HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:30:39 +09:00
Xing Lin
20483f6dc7
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-31 15:17:26 -07:00
Xing Lin
ecafd38c09
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. (#4123) 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02d)

Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-03-31 20:26:09 +00:00
litao
0ecb34f8f6
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) (#4125) 2022-03-31 19:24:05 +09:00
litao
cfca024190 HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:03:27 -07:00
Lei Yang
4f85c9a73b HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:28:38 -07:00
Masatake Iwasaki
f8314cd469 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a244)
 2022-03-28 08:08:59 +00:00
GuoPhilipse
6300fc1fcd
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:24:44 +09:00
qinyuren
55f4baf3e4 HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:05:24 +09:00
Owen O'Malley
1175acbb75 HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 10:24:54 -07:00
Viraj Jasani
712d9bece8
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1c0bc35305)
 2022-03-16 09:40:59 +08:00
Owen O'Malley
c52b97d084 HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:30:29 -07:00
Takanobu Asanuma
52fb9d7ce2 HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>

Cherry-picked from 9c887e5b by Owen O'Malley
 2022-03-14 10:29:37 -07:00
litao
496657c63f HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 
Cherry-picked from 5b05068f by Owen O'Malley
 2022-03-14 10:29:30 -07:00
litao
f9d40ed7b7 HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Cherry-picked from 359b03c8 by Owen O'Malley
 2022-03-14 10:29:11 -07:00
Hui Fei
2479d4ab6c HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang 
Cherry-picked from 264c948e by Owen O'Malley
 2022-03-14 10:29:04 -07:00
Hui Fei
8e129e5b8d HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 
Cherry-picked from 518a212c by Owen O'Malley
 2022-03-14 10:28:55 -07:00
Thinker313
0801fe450e
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-12 19:45:13 +08:00
Chao Sun
b174aaed57 Make upstream aware of 3.3.2 release 2022-03-02 19:10:30 -08:00
litao
74f5f90615 HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 
(cherry picked from commit 6b07c851f3)
 2022-02-26 02:22:28 +09:00
litao
9c0bdc5aea
HDFS-16371. Exclude slow disks when choosing volume (#3753) (#4031) 2022-02-26 02:21:07 +09:00
litao
5601d0848a HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718) 
Authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit 32353eb38a)
 2022-02-25 11:19:27 +09:00
Ayush Saxena
fa30224e95
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
 2022-02-23 11:41:19 +08:00
Chentao Yu
d14a7c6ee5 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:48:13 -08:00
litao
db67952f9f HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0c194f2157)
 2022-02-16 09:45:07 +09:00
Takanobu Asanuma
4c57fb4d6b
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. (#3992) 
(cherry picked from commit 1cd96e8dd8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-16 09:42:43 +09:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
singer-bin
ce7cabb771
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
 2022-02-06 13:36:57 +08:00
daimin
709e617a84
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
 2022-02-06 13:06:35 +08:00
KevinWikant
7171e2190e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)
 2022-01-31 13:19:40 +09:00
KevinWikant
5e2eac6c41
HDFS-16303. Improve handling of datanode lost while decommissioning (#3921) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:18:36 +09:00
secfree
110104da38 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:05:32 +09:00
litao
b5d2e00f81 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:44:03 +09:00
litao
ef1a2b478b HDFS-16398. Reconfig block report parameters for datanode (#3831) 
(cherry picked from commit c2ff39006f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2022-01-26 17:36:17 +09:00
Wei-Chiu Chuang
ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages (#3883) (#3924) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java

Co-authored-by: qinyuren <1476659627@qq.com>
 2022-01-26 11:54:13 +08:00