big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,780 Commits 2 Branches 2 Tags 264 MiB
ff3a88b9c2
Commit Graph

24780 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
huhaiyang
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e528)
 2021-06-01 15:30:50 +08:00
Wei-Chiu Chuang
5cebd02bbe
HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9a0a808338)
 2021-06-01 11:36:27 +08:00
Steve Loughran
bb04b60197
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (#3057) 
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
 2021-05-31 21:49:39 +01:00
zhengchenyu
7feb41b73d
MAPREDUCE-7287. Distcp will delete exists file , If we use "-delete and -update" options and distcp file. (#2852) 
Contributed by zhengchenyu

Change-Id: I61edf9a443c0c6cd5b5dd911901708530cf131ed
 2021-05-28 20:27:00 +01:00
Konstantin V Shvachko
c5535caf6e HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:53:46 -07:00
Takanobu Asanuma
d8689f1a08 Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) 
This reverts commit 0774116756.

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53ff2dfed3)
 2021-05-27 13:17:24 +09:00
Konstantin V Shvachko
5308d44be4 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)
 2021-05-26 15:09:47 -07:00
Akira Ajisaka
e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e)
 2021-05-26 15:48:51 +09:00
Narges Shadab
41025cd585
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 
(cherry picked from commit 3fdeb7435a)
 2021-05-26 14:45:20 +09:00
Jim Brennan
48aa23eb3b YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 
(cherry picked from commit c4581827a9)
 2021-05-25 18:10:24 +00:00
Steve Loughran
464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.

Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
 2021-05-25 12:55:38 +01:00
Akira Ajisaka
ad7071c93b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3)
 2021-05-25 18:29:20 +09:00
Mehakmeet Singh
b82a0fa9e6
HADOOP-17705. S3A to add Config to set AWS region (#3020) 
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.

This is needed when using AWS Private Link, as
the region cannot be automatically determined.

Contributed by Mehakmeet Singh

Change-Id: I4b52f85d7af0ddd56b2b0505ac0124d4fcc67ca0
 2021-05-24 13:13:37 +01:00
Mehakmeet Singh
a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).

Contributed by: Mehakmeet Singh

Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
 2021-05-24 13:04:20 +01:00
Wei-Chiu Chuang
745a2f5ec3
HADOOP-17723. [build] fix the Dockerfile for ARM (#3037) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1576f81dfe)
 2021-05-24 18:20:49 +08:00
Vinayakumar B
dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 
(cherry picked from commit 2bbeae3240)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
 2021-05-24 18:00:38 +08:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Yiqun Lin
ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2021-05-24 10:48:48 +08:00
Szilard Nemeth
6777acb08c YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:36:51 +02:00
Viraj Jasani
87e32fbb8d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548)
 2021-05-22 19:10:28 +09:00
Wei-Chiu Chuang
526dbe4716
HADOOP-17666. Update LICENSE for 3.3.1 (#3011) 
* Inspected the jar files in the produced tarball and updated LICENSE-binary accordingly.

* add LICENSE from hadoop-thirdparty jars.
* remove any dependencies no longer in the tarball.
* Updated the license of thirdparty javascripts and C/C++ files.

Added LICENSE-asio.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/asio-1.10.2/COPYING
Added LICENSE-gmock.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/LICENSE
Added LICENSE-rapidxml.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/rapidxml-1.13/rapidxml/license.txt
Added LICENSE-uriparser2.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser/COPYING
Added LICENSE-tr2.txt, copied from the header of hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/tr2/optional.hpp
Added LICENSE-cJSON.txt, moved from the bottom of LICENSE.txt

* Generated license report for yarn-managed packages.
* Add LICENSE and NOTICES file of jaxb-api.
* Exclude LICENSE-binary-{yarn-applications-catalog-webapp|yarn-ui} from rat report.
These two files are autogenerated.
 2021-05-21 18:15:48 -07:00
Gautham B A
23ada04dac
HADOOP-17509. Parallelize building of dependencies (#2669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6c891c0dd4)
 2021-05-22 01:32:37 +08:00
Wei-Chiu Chuang
5e09d38b25
HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
 2021-05-22 00:31:59 +08:00
Akira Ajisaka
86682a83aa
HADOOP-17663. Remove useless property hadoop.assemblies.version in pom file. (#3025) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3f5a66c158)

 Conflicts:
	hadoop-project/pom.xml
 2021-05-20 10:49:40 +09:00
Jim Brennan
53a1c7653f YARN-10337. Amendment to fix import as in HADOOP-17100 2021-05-19 22:00:55 +00:00
Prabhu Joseph
1b3e4cf9ce YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.

(cherry picked from commit 2bbd00dff4)
 2021-05-19 21:19:05 +00:00
Wei-Chiu Chuang
fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
zhuqi-lucas
7d2eeaecc8 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-05-19 21:24:26 +08:00
Peter Bacsko
a45d6437f5 YARN-10120. In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled. Contributed by Bilwa S T. 2021-05-19 15:07:47 +02:00
Takanobu Asanuma
932a16837a
HADOOP-17224. Install Intel ISA-L library in Dockerfile. (#2537) 
(cherry picked from commit d09e3c929f)
 2021-05-19 20:27:22 +08:00
Peter Bacsko
864b710787 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-19 10:31:57 +02:00
Wei-Chiu Chuang
611ed04aff
HADOOP-17426. Upgrade to hadoop-thirdparty-1.1.0. (#3024) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ibf27246b3eb1b074b938f44f4906db9184466303
(cherry picked from commit 2960d83c25)
 2021-05-19 14:57:45 +09:00
Wei-Chiu Chuang
63392376c9
HDFS-16027. Replace abstract methods with default methods in JournalNodeMXBean. (#3018) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-18 19:49:00 -07:00
Peter Bacsko
27749ce85d YARN-9615. addendum commit, removing accedentially committed patch file 2021-05-18 22:17:20 +02:00
Wei-Chiu Chuang
0df4743284
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de6)
 2021-05-18 11:36:13 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
lujiefsi
137e20cc9b
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 13:53:27 +09:00
litao
2f56e2b54d
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f9896)
 2021-05-17 12:29:11 +08:00
Xiaoyu Yao
3f9c9ccf46
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider.… (#3010) 
* HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334)

(cherry picked from commit 4c5ad57818)
 2021-05-13 16:57:58 -07:00
Wei-Chiu Chuang
1b8eb2654a
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60)
 2021-05-13 20:30:40 +08:00
Aryan Gupta
7142dfec04
HADOOP-17302. Upgrade to jQuery 3.5.1 in hadoop-sls. (#2379) 
Co-authored-by: Aryan Gupta
(cherry picked from commit d60d5fe43d)
 2021-05-13 12:16:24 +08:00
Aryan Gupta
5f147112cf
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 
(cherry picked from commit bbbfa7d415)
 2021-05-13 12:16:20 +08:00
Aryan Gupta
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
 2021-05-13 12:16:17 +08:00
Ayush Saxena
41b39ff292
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 
(cherry picked from commit 960c9ebaea)

Conflicts:
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java
 2021-05-13 12:14:09 +08:00
bilaharith
1c6c3920b6
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2842) 
Contributed by bilaharith
 2021-05-12 18:15:27 +01:00
dependabot[bot]
b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63)
 2021-05-12 10:58:39 +09:00
Peter Bacsko
051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Viraj Jasani
3176ab028b
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b944084b32)
 2021-05-11 12:39:42 +09:00
Borislav Iordanov
f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Ayush Saxena
43db94c7a3 HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:11:09 +05:30