big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,241 Commits 2 Branches 2 Tags 264 MiB
adb673fdaf
Commit Graph

8591 Commits

Author SHA1 Message Date
Masatake Iwasaki
ff13f9ee8b Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57dfd41385dbce5de90620c48a45abb263)
 2022-07-22 02:31:34 +00:00
Viraj Jasani
4ba463069b
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4485) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-23 17:30:01 +08:00
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482) 
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
 2022-06-22 13:09:50 +01:00
Viraj Jasani
7561dbd134
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:44:30 +05:30
KevinWikant
33ab84f2e2 HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6)
 2022-06-20 11:24:45 +09:00
jianghuazhu
18a5e843bc HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:20:34 +08:00
xuzq
ee3ee98ee5 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473)
 2022-06-10 19:02:47 +00:00
Ashutosh Gupta
bdef321d52 HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
 2022-06-09 22:42:04 +09:00
slfan1989
a2f8a9e5d8 HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:59:34 +08:00
Viraj Jasani
516a2a8e44 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:56:07 +08:00
Viraj Jasani
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) (#4405) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-07 06:41:16 +08:00
Samrat
7223a337f6 HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1d)
 2022-06-06 18:02:44 +09:00
Stephen O'Donnell
7d6b133af3 HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2022-06-01 20:54:56 +01:00
Ashutosh Gupta
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785)
 2022-05-26 15:28:27 -07:00
Stephen O'Donnell
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a)
 2022-05-26 10:14:50 -07:00
Wei-Chiu Chuang
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) (#4304) 
(cherry picked from commit cee8c62498)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java

(cherry picked from commit dd79aee635fdc61648e0c87bea1560dc35aee053)

Co-authored-by: caozhiqiang <lfxy@163.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-27 00:50:40 +08:00
Michael Stack
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4347) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 17:02:28 -07:00
jianghuazhu
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. (#4342). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-24 23:47:45 +08:00
Viraj Jasani
ab3a9cedc9 HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:47:18 +08:00
Steve Loughran
fe306ce57e HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:09 +01:00
Michael Stack
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) (#4246) 
Cherry-pick backport of 9ed8d60511
 2022-05-15 21:32:48 -07:00
Viraj Jasani
e16ae55833 HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 09:00:31 +08:00
daimin
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4296) 2022-05-11 07:56:53 +08:00
Ashutosh Gupta
c879230547
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9)
 2022-05-08 08:06:14 +09:00
Ashutosh Gupta
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5b)
 2022-05-08 07:39:17 +09:00
daimin
d711d200fe
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29401c8203)
 2022-05-06 10:35:55 -07:00
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Viraj Jasani
ee450bbbc6
HDFS-16528. Reconfigure slow peer enable for Namenode (#4251) 2022-05-02 08:03:02 +08:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
daimin
b62a460fd9 HDFS-16519. Add throttler to EC reconstruction (#4101) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit aebd55f788)
 2022-04-23 12:47:34 +09:00
litao
9132eeb4dd HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc480)
 2022-04-23 12:15:52 +09:00
S O'Donnell
5e137ac33e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 8ae033d1a3.
 2022-04-20 20:45:17 +01:00
qinyuren
2ff91232bc HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd17374)
 2022-04-20 15:07:38 +09:00
Stephen O'Donnell
8ae033d1a3 HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee0363)
 2022-04-19 11:12:36 +01:00
qinyuren
c913dc3072 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0)
 2022-04-19 13:38:58 +09:00
jianghuazhu
cfe2d8aa79
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e712)
 2022-04-19 11:13:40 +08:00
Ashutosh Gupta
1eb4f9ef04
HDFS-16536. TestOfflineImageViewer fails on branch-3.3 (#4182) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-18 16:05:42 +09:00
Quanlong Huang
9ae903dd1b
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
(cherry picked from commit 35d4c02bcc)
 2022-04-18 10:35:08 +08:00
daimin
0ef1a13f01
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 
(cherry picked from commit c65c383b7e)
 2022-04-14 16:16:36 +08:00
Takanobu Asanuma
52abc9f132 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959)
 2022-04-14 11:50:42 +09:00
qinyuren
07dface36a HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a1)
 2022-04-13 11:48:19 +09:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
singer-bin
26705bbc60
HDFS-16457. Make fs.getspaceused.classname reconfigurable (apache#4069) (#4156) 2022-04-11 14:59:34 +09:00
Takanobu Asanuma
30afe7ca20 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 
(cherry picked from commit 37650ced81)
 2022-04-08 18:39:26 +09:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00
Stephen O'Donnell
bd0dbf319a HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf)
 2022-04-06 11:54:58 +01:00
Viraj Jasani
b2eee14f2e HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7c20602b17)
 2022-04-06 18:55:09 +09:00
Viraj Jasani
a6fb77f7eb HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 
(cherry picked from commit 278568203b)
 2022-04-06 18:40:07 +09:00
wangzhaohui
0e621c890d HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:30:39 +09:00
Xing Lin
20483f6dc7
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-31 15:17:26 -07:00
Xing Lin
ecafd38c09
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. (#4123) 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02d)

Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-03-31 20:26:09 +00:00
litao
0ecb34f8f6
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) (#4125) 2022-03-31 19:24:05 +09:00
litao
cfca024190 HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:03:27 -07:00
Lei Yang
4f85c9a73b HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:28:38 -07:00
Masatake Iwasaki
f8314cd469 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a244)
 2022-03-28 08:08:59 +00:00
GuoPhilipse
6300fc1fcd
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:24:44 +09:00
qinyuren
55f4baf3e4 HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:05:24 +09:00
Owen O'Malley
1175acbb75 HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 10:24:54 -07:00
Viraj Jasani
712d9bece8
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1c0bc35305)
 2022-03-16 09:40:59 +08:00
Takanobu Asanuma
52fb9d7ce2 HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>

Cherry-picked from 9c887e5b by Owen O'Malley
 2022-03-14 10:29:37 -07:00
litao
496657c63f HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 
Cherry-picked from 5b05068f by Owen O'Malley
 2022-03-14 10:29:30 -07:00
litao
f9d40ed7b7 HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Cherry-picked from 359b03c8 by Owen O'Malley
 2022-03-14 10:29:11 -07:00
Thinker313
0801fe450e
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-12 19:45:13 +08:00
Chao Sun
b174aaed57 Make upstream aware of 3.3.2 release 2022-03-02 19:10:30 -08:00
litao
74f5f90615 HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 
(cherry picked from commit 6b07c851f3)
 2022-02-26 02:22:28 +09:00
litao
9c0bdc5aea
HDFS-16371. Exclude slow disks when choosing volume (#3753) (#4031) 2022-02-26 02:21:07 +09:00
litao
5601d0848a HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718) 
Authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit 32353eb38a)
 2022-02-25 11:19:27 +09:00
Ayush Saxena
fa30224e95
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
 2022-02-23 11:41:19 +08:00
Chentao Yu
d14a7c6ee5 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:48:13 -08:00
litao
db67952f9f HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0c194f2157)
 2022-02-16 09:45:07 +09:00
Takanobu Asanuma
4c57fb4d6b
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. (#3992) 
(cherry picked from commit 1cd96e8dd8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-16 09:42:43 +09:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
singer-bin
ce7cabb771
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
 2022-02-06 13:36:57 +08:00
KevinWikant
7171e2190e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)
 2022-01-31 13:19:40 +09:00
KevinWikant
5e2eac6c41
HDFS-16303. Improve handling of datanode lost while decommissioning (#3921) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:18:36 +09:00
secfree
110104da38 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:05:32 +09:00
litao
b5d2e00f81 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:44:03 +09:00
litao
ef1a2b478b HDFS-16398. Reconfig block report parameters for datanode (#3831) 
(cherry picked from commit c2ff39006f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2022-01-26 17:36:17 +09:00
Wei-Chiu Chuang
ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages (#3883) (#3924) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java

Co-authored-by: qinyuren <1476659627@qq.com>
 2022-01-26 11:54:13 +08:00
Bryan Beaudreault
bd13d73334 HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 
(cherry picked from commit 94b884ae55)
 2022-01-25 11:43:47 +00:00
daimin
728ed10a7c
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
 2022-01-25 09:58:20 +08:00
liubingxing
d6ff60df65
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
 2022-01-20 18:47:00 +08:00
litao
f9c0bc094a HDFS-16399. Reconfig cache report parameters for datanode (#3841) 
(cherry picked from commit e355646330)
 2022-01-19 18:43:15 +09:00
litao
11fe5279b0 HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit f02374df92)
 2022-01-19 18:42:48 +09:00
litao
cdaf4d89f9 HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 52ec65fd10)
 2022-01-19 18:40:41 +09:00
Viraj Jasani
831c11c47a HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit b038042ece)
 2022-01-19 16:01:00 +09:00
qinyuren
1c71d6e9fe HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:02 +09:00
Xiangyi Zhu
b5e7f59e53
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 23:18:05 +08:00
Jackson Wang
926222a0d0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:40:58 +09:00
Ashutosh Gupta
6b83fe4a00 HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:00:23 -08:00
liever18
3a82899493 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 15:46:21 +00:00
Wei-Chiu Chuang
350b51f287 Make upstream aware of 3.3.1 release 2022-01-04 14:48:49 -08:00
Ashutosh Gupta
bad3a0964c HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:25:56 +09:00
jianghuazhu
b623acff5a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:25:00 +09:00
jianghuazhu
a4496a0187 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 11:32:01 +00:00
litao
e60b8a0358 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:51:40 +09:00
litao
75af1d67e4 HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2021-12-16 13:31:17 +09:00
Ayush Saxena
37ced452d9 HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:23:24 +09:00
litao
8559ba46c2 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2021-12-14 11:53:46 +09:00
GuoPhilipse
2720479988 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2021-12-10 16:33:33 +09:00