big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,052 Commits 2 Branches 2 Tags 264 MiB
b737869e01
Commit Graph

26052 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gautham B A
dc5460d525
YARN-11078. Set env vars in a cross platform compatible way (#4432) 
* Maven runs the ember build script.
  The environment variable TMPDIR was
  set as per bash syntax.
* This failed on Windows since the
  Windows command prompt doesn't
  support bash syntax.
* We're now detecting the OS and
  setting a Maven property
  "emberBuildScript" in a cross
  platform compatible way.
 2022-06-15 15:29:55 +05:30
Gautham B A
5a40224b53
HDFS-16469. Locate protoc-gen-hrpc across platforms (#4434) 
* We use the TARGET_FILE CMake
  generator expression to get
  the location of the
  protoc-gen-hrpc CMake target.
 2022-06-15 15:28:10 +05:30
9uapaw
75bc6cfced YARN-11176. Refactor TestAggregatedLogDeletionService. Contributed by Szilard Nemeth. 2022-06-14 16:14:44 +02:00
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
9uapaw
c9a174a260 YARN-11175. Refactor LogAggregationFileControllerFactory 2022-06-13 13:58:13 +02:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
jianghuazhu
7f5a34dfaa
HDFS-16621.Remove unused JNStorage#getCurrentDir() (#4404). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-10 12:12:41 +05:30
Gautham B A
d557c44d94
HDFS-16463. Make dirent cross platform compatible (#4370) 
* jnihelper.c in HDFS native client uses
  dirent.h. This header file isn't available
  on Windows.
* This PR provides a cross platform
  compatible implementation for dirent
  under the XPlatform library.
 2022-06-10 09:59:07 +05:30
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
Steve Loughran
985161f6ec
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi (#4417) 
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.

Contributed by Steve Loughran
 2022-06-09 13:10:46 +01:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
slfan1989
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation (#4396) 2022-06-08 17:34:43 -07:00
monthonk
5ac55b405d
HADOOP-12020. Add s3a storage class option fs.s3a.create.storage.class (#3877) 
Adds a new option fs.s3a.create.storage.class which can
be used to set the storage class for files created in AWS S3.
Consult the documentation for details and instructions on how
disable the relevant tests when testing against third-party
stores.

Contributed by Monthon Klongklaew
 2022-06-08 19:05:17 +01:00
Viraj Jasani
0a65883c50
HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:53:45 +08:00
Steve Loughran
a234d00c1c
HADOOP-18275. Update os-maven-plugin to 1.7.0 (#4397) 
Contributed by Steve Loughran
 2022-06-06 13:17:32 +01:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
Renukaprasad C
7bd4ac3ce0
HDFS-16563. Namenode WebUI prints sensitive information on Token expiry (#4241) 
Contributed by Renukaprasad C
 2022-06-03 18:37:27 +01:00
Gautham B A
a406f49bc0
HDFS-16602. Use "defined" directive along with #if (#4371) 
* The #if directive expects a
  boolean expression. This PR
  uses the more appropriate
  "defined" directive.
 2022-06-03 22:05:20 +05:30
slfan1989
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation (#4341) 2022-06-02 12:54:29 -07:00
Gautham B A
7ac271307c
HADOOP-18274. Use CMake 3.19.0 in Debian 10 (#4395) 2022-06-02 11:16:05 -07:00
Gautham B A
712e009172
HADOOP-18268. Install maven from Apache archives (#4373) 
* The remote server hosting maven
  was temporarily unavailable causing
  the Hadoop Jenkins CI to fail.
* Thus, we're switching to use a CDN
  to mitigate such issues.
 2022-06-02 17:05:18 +05:30
Stephen O'Donnell
34a973a90e
HDFS-16610. Make fsck read timeout configurable (#4384) 2022-06-01 20:36:01 +01:00
Gautham B A
4496e2daf6
HDFS-16604. Install gtest via FetchContent_Declare in CMake (#4374) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-01 13:45:44 +09:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
Ashutosh Gupta
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project (#4381). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 23:17:11 +05:30
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
Ashutosh Gupta
21fa693d38
HADOOP-18238. Fix reentrancy check in SFTPFileSystem.close() (#4330) 
Contributed by Ashutosh Gupta
 2022-05-30 17:34:06 +01:00
GuoPhilipse
ba6520f67f
HADOOP-18269. Misleading method name in DistCpOptions.(#4216) 
Contributed by guophilipse
 2022-05-30 14:02:47 +01:00
HanleyYang
835f39cefc
HDFS-15878. RBF: Fix TestRouterWebHDFSContractCreate#testSyncable. (#4340). Contributed by Hanley Yang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:56:07 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Ashutosh Gupta
bee538f785
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-27 05:18:15 +08:00
Stephen O'Donnell
297f0f6d6a
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-05-27 01:11:10 +08:00
Ashutosh Gupta
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-26 15:51:55 +09:00
Rishabh Sharma
0be1fde962
HDFS-16561. Handle error returned by strtol 
* strtol is used in hdfs-chmod.cc. The call
  to strtol could error out when an invalid
  input is provided.
* This PR handles the error given out by
   strtol.
 2022-05-26 11:09:49 +05:30
Masatake Iwasaki
6b331dde31
HADOOP-18251. Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py. (#4344) 2022-05-26 12:23:30 +09:00
Ashutosh Gupta
84b0455cf8
HADOOP-18240. Upgrade Yetus to 0.14.0 (#4328) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-25 17:31:18 +09:00
slfan1989
678768035d
YARN-11137. Improve log message in FederationClientInterceptor (#4336) 2022-05-24 17:21:04 -07:00
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
Owen O'Malley
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 2022-05-24 14:07:19 +09:00
Michael Stack
5ab747222b
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338) 
* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
 2022-05-23 08:45:08 -07:00
jianghuazhu
4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-22 21:59:38 +08:00
Viraj Jasani
93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:28:59 +08:00
slfan1989
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation (#4317) 2022-05-20 13:19:42 -07:00