big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,919 Commits 2 Branches 2 Tags 264 MiB
477b67a335
Commit Graph

9657 Commits

Author SHA1 Message Date
Ashutosh Gupta
efc2761d32
HDFS-16635.Fixed javadoc error in Java 11 (#4451) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-20 11:14:31 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00
xuzq
4893f00395
HDFS-16600. Fix deadlock of fine-grain lock for FsDatastImpl of DataNode. (#4367). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-17 22:05:33 +08:00
caozhiqiang
9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu
6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:18:58 +08:00
章锡平
f8c7e67fdc
HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. 2022-06-15 21:16:24 +08:00
Gautham B A
5a40224b53
HDFS-16469. Locate protoc-gen-hrpc across platforms (#4434) 
* We use the TARGET_FILE CMake
  generator expression to get
  the location of the
  protoc-gen-hrpc CMake target.
 2022-06-15 15:28:10 +05:30
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
jianghuazhu
7f5a34dfaa
HDFS-16621.Remove unused JNStorage#getCurrentDir() (#4404). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-10 12:12:41 +05:30
Gautham B A
d557c44d94
HDFS-16463. Make dirent cross platform compatible (#4370) 
* jnihelper.c in HDFS native client uses
  dirent.h. This header file isn't available
  on Windows.
* This PR provides a cross platform
  compatible implementation for dirent
  under the XPlatform library.
 2022-06-10 09:59:07 +05:30
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
Viraj Jasani
0a65883c50
HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:53:45 +08:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
Gautham B A
a406f49bc0
HDFS-16602. Use "defined" directive along with #if (#4371) 
* The #if directive expects a
  boolean expression. This PR
  uses the more appropriate
  "defined" directive.
 2022-06-03 22:05:20 +05:30
Stephen O'Donnell
34a973a90e
HDFS-16610. Make fsck read timeout configurable (#4384) 2022-06-01 20:36:01 +01:00
Gautham B A
4496e2daf6
HDFS-16604. Install gtest via FetchContent_Declare in CMake (#4374) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-01 13:45:44 +09:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Ashutosh Gupta
bee538f785
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-27 05:18:15 +08:00
Stephen O'Donnell
297f0f6d6a
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-05-27 01:11:10 +08:00
Rishabh Sharma
0be1fde962
HDFS-16561. Handle error returned by strtol 
* strtol is used in hdfs-chmod.cc. The call
  to strtol could error out when an invalid
  input is provided.
* This PR handles the error given out by
   strtol.
 2022-05-26 11:09:49 +05:30
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
Michael Stack
5ab747222b
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338) 
* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
 2022-05-23 08:45:08 -07:00
jianghuazhu
4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-22 21:59:38 +08:00
Viraj Jasani
93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:28:59 +08:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
Ashutosh Gupta
fb910bd906
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 02:53:14 +09:00
Steve Loughran
6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:54 +01:00
Hui Fei
dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen
680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
singer-bin
d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Gautham B A
1350539f2d
HDFS-16465. Remove redundant strings.h inclusions (#4279) 
* Remove redundant strings.h inclusions

* strings.h was included in a bunch of
  C/C++ files and were redundant.
* Also, strings.h is not available on
  Windows and thus isn't cross-platform
  compatible.

* Build for all platforms in CI

* Revert "Build for all platforms in CI"

This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb.

* Debug failure on Centos 8

* Skipping pipeline run on
  Centos 7 to debug the
  failure on Centos 8.

* Revert "Debug failure on Centos 8"

This reverts commit e365e34d6fab9df88f4df622910ddb28a8c8796f.
 2022-05-11 23:04:22 +05:30
Viraj Jasani
6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
daimin
d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
Ashutosh Gupta
01027e52a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:05:34 +09:00
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
daimin
29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Gautham B A
d346be9bd2
HDFS-16564. Use uint32_t for hdfs_find (#4245) 
* hdfs_find uses u_int32_t type for
  storing the value for the max-depth
  command line argument.
* The type u_int32_t isn't standard,
  isn't available on Windows and thus
  breaks cross-platform compatibility.
  We need to replace this with uint32_t
  which is available on all platforms since
  it's part of the C++ standard.
 2022-05-04 20:19:08 +05:30