big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,206 Commits 2 Branches 2 Tags 264 MiB
be70bbb4be
Commit Graph

9735 Commits

Author SHA1 Message Date
jianghuazhu
900682e712
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 11:21:51 +08:00
Quanlong Huang
35d4c02bcc
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
 2022-04-18 10:32:29 +08:00
Stephen O'Donnell
dbeeee0363
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 2022-04-17 20:05:11 +08:00
He Xiaoqiao
cb975c3df6
HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141) Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-04-17 19:21:27 +08:00
Renukaprasad C
f14f305051
HDFS-16526. Add metrics for slow DataNode (#4162) 2022-04-15 21:37:05 +05:30
caozhiqiang
cee8c62498
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) 2022-04-14 18:42:39 +09:00
daimin
c65c383b7e
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 2022-04-14 11:07:06 +08:00
Takanobu Asanuma
2efab92959
HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 2022-04-14 11:23:38 +09:00
Steve Loughran
a7b4e8f03e
HDFS-14478. Add libhdfs APIs for openFile (#4166) 
Contributed by Sahil Takiar
 2022-04-13 14:15:27 +01:00
qinyuren
45394433a1
HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 2022-04-13 11:27:37 +09:00
Gautham B A
e89fd9645b
HDFS-16474. Make HDFS tail tool cross platform (#4157) 
* The source files for hdfs_tail
  uses getopt for parsing the
  command line arguments.
* getopt is available only on
  Linux and thus, isn't cross
  platform.
* We need to replace getopt
  with boost::program_options
  to make these tools cross
  platform.
 2022-04-12 20:31:24 +05:30
Gautham B A
d5e97fe4d6
HDFS-16473. Make HDFS stat tool cross platform (#4145) 
* The source files for hdfs_stat
  uses getopt for parsing the
  command line arguments.
* getopt is available only on
  Linux and thus, isn't cross platform.
* We need to replace getopt with
  boost::program_options to make
  this tool cross platform.
 2022-04-08 23:06:39 +05:30
Takanobu Asanuma
37650ced81 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 2022-04-08 18:38:40 +09:00
litao
5412fbf6d4
HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001) 2022-04-08 12:26:39 +08:00
singer-bin
807a428b55
HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069) 2022-04-08 10:27:35 +09:00
Viraj Jasani
f70935522b
HADOOP-18188. Support touch command for directory (#4135) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-07 17:29:45 +09:00
Viraj Jasani
7c20602b17
HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 18:17:02 +09:00
wangzhaohui
61bbdfd3a7
HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 17:29:43 +09:00
Stephen O'Donnell
bbfe3500cf
HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 2022-04-06 08:49:52 +01:00
Gautham B A
4ef1d3eef9
HDFS-16472. Make HDFS setrep tool cross platform (#4130) 
* The source files for hdfs_setrep
   uses getopt for parsing the
   command line arguments.
* getopt is available only on Linux
   and thus, isn't cross platform.
* We need to replace getopt
  with boost::program_options
  to make this tool cross platform.
 2022-04-05 22:59:11 +05:30
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
Xing Lin
15a5ea2c95
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
 2022-03-31 21:59:09 +00:00
litao
9a4dddd640
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:01:48 -07:00
He Xiaoqiao
2bf78e2416
HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mode. (#4085). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-31 14:00:38 +08:00
litao
ac50657c37
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) 2022-03-31 00:51:17 +09:00
He Xiaoqiao
6eea28c3f3
HDFS-16498. Fix NPE for checkBlockReportLease #4057. Contributed by tomscut. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-30 14:56:04 +08:00
Lei Yang
a9b43966c0
HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:15:28 -07:00
Masatake Iwasaki
0fbd96a244 Make upstream aware of 3.2.3 release. 2022-03-28 08:07:50 +00:00
GuoPhilipse
046a6204b4
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-27 22:23:48 +09:00
litao
565e848d88
HDFS-16434. Add opname to read/write lock for remaining operations (#3915) 2022-03-25 20:19:01 +09:00
qinyuren
45ce1cce50
HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
 2022-03-23 15:03:17 +09:00
He Xiaoqiao
ef8bff0df9
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-23 11:32:12 +08:00
He Xiaoqiao
26ba3846cc
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang." 
This reverts commit 88975496d8.
 2022-03-23 11:01:35 +08:00
Gautham B A
81879eb9cb
HDFS-16471. Make HDFS ls tool cross platform (#4086) 
* The source files for hdfs_ls
   uses getopt for parsing the
   command line arguments.
* getopt is available only on
   Linux and thus, isn't cross
   platform.
* Thus, we need to replace
   getopt with
   boost::program_options to
   make this tool cross platform.
 2022-03-22 23:12:00 +05:30
He Xiaoqiao
88975496d8
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-22 22:28:09 +08:00
Owen O'Malley
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 09:27:35 -07:00
litao
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-03-21 12:40:51 +05:30
He Xiaoqiao
62d59e516e
HDFS-16504. Add parameter for NameNode to process getBloks request. (#4068). Contributed by Max Xie. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-20 14:01:45 +08:00
Gautham B A
a631f45a99
HDFS-16470. Make HDFS find tool cross platform (#4076) 
* The source files for hdfs_find uses
   getopt for parsing the command
   line arguments. getopt is available
   only on Linux and thus, isn't cross
   platform.
* Thus, we need to replace getopt
   with boost::program_options to
   make hdfs_find cross platform.
 2022-03-18 21:41:01 +05:30
litao
7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058) 2022-03-17 12:04:48 +08:00
He Xiaoqiao
a237526988
HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by JiangHua Zhu. 2022-03-16 20:43:38 +08:00
Viraj Jasani
1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-16 09:32:29 +08:00
Xing Lin
8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 11:29:48 -07:00
Owen O'Malley
7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
He Xiaoqiao
a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
 2022-03-12 18:40:09 +08:00
Stephen O'Donnell
49e4e4d0a5
HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error (#4051) 2022-03-08 11:07:11 +00:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Gautham B A
8f07081789
HDFS-16462. Make HDFS get tool cross platform (#4003) 2022-03-05 23:05:14 +05:30
Gautham B A
19561846df
HADOOP-18151. Switch the baseurl for Centos 8 (#4047) 
* Centos 8 has reached its
  End-of-Life and thus its
  packages are no longer
  accessible from
  mirror.centos.org.
* This PR switches the baseurl
  to vault.centos.org where
  the packages are archived.
 2022-03-05 07:11:22 +05:30
Viraj Jasani
278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 2022-03-04 22:17:48 +05:30
Chao Sun
f800b65b40 Make upstream aware of 3.3.2 release 2022-03-02 19:14:50 -08:00
litao
9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041) 2022-03-02 11:25:50 +08:00
SevenAddSix
5eab9719cb
HDFS-16480. Fix typo: indicies -> indices (#4020) 2022-02-28 21:12:18 +08:00
Hui Fei
b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998)" (#4038) 2022-02-28 21:05:19 +08:00
litao
8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998) 2022-02-28 21:03:59 +08:00
litao
6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 2022-02-25 01:23:20 +09:00
Viraj Jasani
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Neil
2f448acf39
HDFS-16459. RBF: register RBFMetrics in MetricsSystem for promethuessink (#3999) 2022-02-21 11:32:34 -08:00
Chentao Yu
19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
YulongZ
48bef285a8
HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971) 2022-02-15 08:44:17 -08:00
litao
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 13:40:46 +09:00
Neil
c777142ed3
RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. (#3965) 2022-02-10 09:17:30 -08:00
Neil
390967f1f0
RBF: Add search box in Router's tab-mounttable web page (#3968) 2022-02-09 09:17:23 -08:00
secfree
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847) 2022-02-09 10:07:05 +08:00
YulongZ
3e7a7c3c4a
HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878) 2022-02-08 08:40:57 -08:00
singer-bin
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 2022-02-06 13:05:57 +08:00
litao
b5b07af605
HDFS-16435. Remove no need TODO comment for ObserverReadProxyProvider (#3912). Contributed by tomscut. 
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-03 08:21:06 +05:30
Gautham B A
ed44662968
HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945) 2022-02-02 11:12:17 +05:30
KevinWikant
089e06de21
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:04:32 +09:00
litao
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web (#3943) 2022-01-30 16:13:58 +08:00
secfree
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 17:04:11 +09:00
Li MingXiang
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00
litao
6136d630a3
HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 2022-01-27 13:42:44 +09:00
litao
c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode (#3831) 2022-01-26 17:24:09 +09:00
jianghuazhu
8c7c49d877
HDFS-16401.Remove the worthless DatasetVolumeChecker#numAsyncDatasetChecks. (#3838) 2022-01-25 21:51:17 +08:00
Bryan Beaudreault
94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 2022-01-25 11:42:35 +00:00
Thinker313
43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-25 15:26:18 +08:00
daimin
d69938994e
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-01-25 13:02:37 +08:00
daimin
5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-01-24 12:04:26 +05:30
litao
15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. (#3839). Contributed by tomscut. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-24 14:26:30 +08:00
qinyuren
db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages (#3883) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-19 17:59:42 +09:00
litao
e355646330
HDFS-16399. Reconfig cache report parameters for datanode (#3841) 2022-01-19 15:10:39 +09:00
qinyuren
fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 2022-01-19 13:42:33 +09:00
Jackson Wang
d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 22:38:11 +09:00
litao
f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 14:48:10 +09:00
Xiangyi Zhu
034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063). Contributed by Xiangyi Zhu. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-13 12:56:19 +08:00
Gautham B A
271ac0e091
HDFS-16419. Make HDFS data transfer tools cross platform (#3873) 2022-01-12 19:57:27 +05:30
luoyuan3471
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Felix Nguyen
b88d5ae1c6
HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871) 2022-01-10 18:28:20 +08:00
Viraj Jasani
f64fda0f00
HADOOP-18055. Async Profiler endpoint for Hadoop daemons (#3824) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-06 17:56:49 +08:00
litao
46b02788bd
HDFS-16371. Exclude slow disks when choosing volume (#3753) 2022-01-06 09:31:00 +09:00
Ashutosh Gupta
43e5218a86
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 15:51:14 +00:00
liever18
e1d0aa9ee5
HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 2022-01-05 15:42:58 +00:00
Gautham B A
c3006be516
HDFS-16407. Make hdfs_du tool cross platform (#3848) 2022-01-04 22:29:54 +05:30
Ashutosh Gupta
9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 13:25:16 +09:00
Ayush Saxena
61d424f385
HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-01-04 07:50:01 +05:30
Symious
b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline (#3704) 2021-12-30 00:27:11 +09:00
jianghuazhu
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 17:24:12 +09:00
KevinWikant
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 22:59:08 +09:00
secfree
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
wangzhaohui
8d251bd629
HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 2021-12-21 11:28:57 +00:00
Jackson Wang
9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. (#3803) 
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
 2021-12-21 11:04:02 +09:00
jianghuazhu
746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 2021-12-20 11:28:55 +00:00
secfree
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
liubingxing
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-17 13:31:28 +08:00
Wei-Chiu Chuang
bdec546671
Revert "HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798)" 
This reverts commit a4557f9ed9.
 2021-12-16 21:27:08 +08:00
litao
22f5e1885d
HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-16 13:49:50 +09:00
Tamás Pénzes
a4557f9ed9
HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798) 2021-12-16 12:38:42 +08:00
litao
10cdde59b4
HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 2021-12-16 13:29:32 +09:00
litao
59c650802d
HDFS-16378. Add datanode address to BlockReportLeaseManager logs (#3786). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-15 11:16:32 +08:00
Ayush Saxena
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:21:51 +09:00
litao
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 2021-12-14 11:19:06 +09:00
Akira Ajisaka
9b9e2ef87f
HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:36:31 +09:00
Gautham B A
6dddbd42ed
HDFS-16285. Make HDFS ownership tools cross platform (#3588) 2021-12-09 10:17:41 +05:30
qinyuren
35556ea043
HDFS-16333. fix balancer bug when transfer an EC block (#3679) 2021-12-09 13:24:09 +09:00
PHILO-HE
1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761) 2021-12-08 17:52:40 +05:30
GuoPhilipse
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode (#3661) 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 21:21:48 +09:00
lzx404243
7ba91c1235
HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 20:44:18 +09:00
GuoPhilipse
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-08 16:34:54 +05:30
Masatake Iwasaki
e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740) 2021-12-07 21:39:04 +09:00
GuoPhilipse
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem (#3713). Contributed by guophilipse. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-07 14:19:27 +05:30
GuoPhilipse
1509ea5de2
HDFS-16338. Correct fsimage error configuration message (#3684). Contributed by guophilipse. 
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-06 14:00:50 +05:30
Takanobu Asanuma
e8e69de106 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 2021-12-06 10:44:36 +09:00
litao
c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo (#3747). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 22:54:44 +05:30
GuoPhilipse
2f55a6cced
HDFS-16357. Fix log format in DFSUtilClient (#3729). Contributed by guophilipse. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 18:35:05 +05:30
Ayush Saxena
cab7086fbc
HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745). Contributed by Ayush Saxena. 
Reviewed-by: litao <tomleescut@gmail.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-12-04 10:54:29 +05:30
huhaiyang
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-04 02:19:36 +09:00
bitterfox
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-03 23:30:13 +09:00
wangzhaohui
0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 (#3742) 2021-12-03 16:51:04 +05:30
litao
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-03 14:12:05 +09:00
Viraj Jasani
0c62a514f7
HDFS-16358. HttpFS implementation for getSnapshotDiffReportListing (#3730) 2021-12-03 13:25:32 +09:00
litao
196935a8d2
HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731). Contributed by tomscut. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-02 19:10:35 +05:30
litao
1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand (#3732). Contributed by tomscut. 2021-12-01 12:48:21 +05:30
litao
829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-29 17:48:10 +09:00
litao
86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-26 19:52:14 +09:00
Takanobu Asanuma
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-25 14:05:04 +09:00
Viraj Jasani
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 2021-11-25 10:23:04 +08:00
huhaiyang
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Viraj Jasani
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 22:54:11 +09:00
Mukul Kumar Singh
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 2021-11-23 13:36:14 +00:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md (#3681) 2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu
3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629) 2021-11-12 09:52:02 +08:00
litao
1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 10:23:25 +09:00
litao
2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:22:32 +09:00
litao
ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:16:45 +09:00
jianghuazhu
b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao
cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-09 17:06:28 +09:00
GuoPhilipse
9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability 2021-11-08 10:21:39 -08:00
Gautham B A
0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625) 2021-11-08 08:58:07 -08:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Gautham B A
1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617) 2021-11-04 09:19:11 -07:00
jianghuazhu
72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 2021-11-04 16:43:25 +09:00
jianghuazhu
18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602) 2021-11-04 09:51:11 +08:00
litao
359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
daimin
a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Stephen O'Donnell
2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu
618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-02 00:55:50 +09:00
jianghuazhu
0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600) 
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600). Contributed by JiangHua Zhu.
 2021-10-31 08:48:52 +05:30
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
litao
6aa664c624
HDFS-16279. Print detail datanode info when process first storage report (#3564) 2021-10-29 10:27:00 +09:00
litao
c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
daimin
6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:12:13 +05:30
Ayush Saxena
f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin." 
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc.
 2021-10-22 21:10:38 +05:30
daimin
7b913bc6bc
Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:09:37 +05:30
litao
2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568). Contributed by tomscut. 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:21:30 +05:30
GuoPhilipse
1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 07:59:27 +05:30
Michael Kuchenbecker
a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
Gautham B A
a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform (#3563) 2021-10-21 10:24:33 -07:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
daimin
5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
Xiangyi Zhu
d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 13:53:41 +09:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
litao
d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554) 
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
 2021-10-15 22:41:38 +05:30
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
LeonGao
7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Gautham B A
107fe227eb
HDFS-16267. Make hdfs_df tool cross platform (#3542) 2021-10-13 09:11:21 -07:00
jianghuazhu
ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497) 2021-10-13 17:21:03 +08:00
Gautham B A
a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532) 2021-10-11 18:25:28 -07:00