big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,507 Commits 2 Branches 2 Tags 264 MiB
4825c7c28d
Commit Graph

24507 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viraj Jasani
4825c7c28d MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa)
 2021-06-21 16:47:37 +09:00
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
 2021-06-21 10:28:05 +09:00
litao
7d202e8365 HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:24:03 +05:30
Viraj Jasani
ef3a2a3b3b HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:52 +05:30
Ahmed Hussein
92ade1f6f9 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:49 +05:30
litao
8d241482bd HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:38 +05:30
Hui Fei
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3113) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 14:41:02 +08:00
Wei-Chiu Chuang
577b96ef42
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548)
 2021-06-17 11:39:34 +08:00
Takanobu Asanuma
25138c98bf HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129)
 2021-06-17 10:00:36 +09:00
Ayush Saxena
3e03ecd680 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:54:36 +05:30
Jim Brennan
57ee4238eb YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit 9a6a11c452)
 2021-06-15 19:30:16 +00:00
Takanobu Asanuma
546c66b812 HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682)
 2021-06-15 17:22:34 +09:00
Ahmed Hussein
840612b3b1 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 
(cherry picked from commit 747883af4d)
 2021-06-14 14:37:27 +00:00
Takanobu Asanuma
10000f162d HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. (#3090) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580)
 2021-06-14 10:21:30 +09:00
Petre Bogdan Stolojan
254d943126
HADOOP-17547 Magic committer to downgrade abort in cleanup if list uploads fails with access denied (#3051) 
Contributed by Bogdan Stolojan

Change-Id: I32d6dc4f72087783a3ea12473d11690ac14fe3cb
 2021-06-12 17:46:11 +01:00
litao
52d055a7eb HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0)
 2021-06-11 17:33:06 +09:00
Mingliang Liu
91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 8c0f948054.
 2021-06-11 00:35:41 -07:00
Akira Ajisaka
19c969981d
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. (#3068) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9a17f97133)
 2021-06-10 14:33:41 +09:00
Viraj Jasani
0d6939ffc4
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca27744770)
 2021-06-09 10:38:38 +08:00
Wei-Chiu Chuang
83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. (#3054) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-08 19:32:34 -07:00
Viraj Jasani
8f0ba9ee1b
HADOOP-17725. Improve error message for token providers in ABFS (#3041) 
Contributed by Viraj Jasani.
 2021-06-08 22:05:01 +01:00
Steve Loughran
4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.

Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
 2021-06-08 22:05:00 +01:00
litao
02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a)
 2021-06-08 16:59:59 +09:00
litao
46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 (#3021) 2021-06-08 00:10:37 -07:00
Akira Ajisaka
37516726d7 HDFS-16050. Some dynamometer tests fail. (#3079) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 57a3613e5d)
 2021-06-07 15:03:06 +09:00
Hideyuki Furue
8ca57fc9f5
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9c7b8cf54e)
 2021-06-06 07:29:22 +09:00
July
8c0f948054
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:37:24 -07:00
huhaiyang
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e528)
 2021-06-01 15:30:50 +08:00
Wei-Chiu Chuang
5cebd02bbe
HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9a0a808338)
 2021-06-01 11:36:27 +08:00
Steve Loughran
bb04b60197
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (#3057) 
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
 2021-05-31 21:49:39 +01:00
zhengchenyu
7feb41b73d
MAPREDUCE-7287. Distcp will delete exists file , If we use "-delete and -update" options and distcp file. (#2852) 
Contributed by zhengchenyu

Change-Id: I61edf9a443c0c6cd5b5dd911901708530cf131ed
 2021-05-28 20:27:00 +01:00
Konstantin V Shvachko
c5535caf6e HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:53:46 -07:00
Takanobu Asanuma
d8689f1a08 Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) 
This reverts commit 0774116756.

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53ff2dfed3)
 2021-05-27 13:17:24 +09:00
Konstantin V Shvachko
5308d44be4 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)
 2021-05-26 15:09:47 -07:00
Akira Ajisaka
e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e)
 2021-05-26 15:48:51 +09:00
Narges Shadab
41025cd585
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 
(cherry picked from commit 3fdeb7435a)
 2021-05-26 14:45:20 +09:00
Jim Brennan
48aa23eb3b YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 
(cherry picked from commit c4581827a9)
 2021-05-25 18:10:24 +00:00
Steve Loughran
464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.

Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
 2021-05-25 12:55:38 +01:00
Akira Ajisaka
ad7071c93b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3)
 2021-05-25 18:29:20 +09:00
Mehakmeet Singh
b82a0fa9e6
HADOOP-17705. S3A to add Config to set AWS region (#3020) 
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.

This is needed when using AWS Private Link, as
the region cannot be automatically determined.

Contributed by Mehakmeet Singh

Change-Id: I4b52f85d7af0ddd56b2b0505ac0124d4fcc67ca0
 2021-05-24 13:13:37 +01:00
Mehakmeet Singh
a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).

Contributed by: Mehakmeet Singh

Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
 2021-05-24 13:04:20 +01:00
Wei-Chiu Chuang
745a2f5ec3
HADOOP-17723. [build] fix the Dockerfile for ARM (#3037) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1576f81dfe)
 2021-05-24 18:20:49 +08:00
Vinayakumar B
dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 
(cherry picked from commit 2bbeae3240)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
 2021-05-24 18:00:38 +08:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Yiqun Lin
ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2021-05-24 10:48:48 +08:00
Szilard Nemeth
6777acb08c YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:36:51 +02:00
Viraj Jasani
87e32fbb8d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548)
 2021-05-22 19:10:28 +09:00
Wei-Chiu Chuang
526dbe4716
HADOOP-17666. Update LICENSE for 3.3.1 (#3011) 
* Inspected the jar files in the produced tarball and updated LICENSE-binary accordingly.

* add LICENSE from hadoop-thirdparty jars.
* remove any dependencies no longer in the tarball.
* Updated the license of thirdparty javascripts and C/C++ files.

Added LICENSE-asio.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/asio-1.10.2/COPYING
Added LICENSE-gmock.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/LICENSE
Added LICENSE-rapidxml.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/rapidxml-1.13/rapidxml/license.txt
Added LICENSE-uriparser2.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser/COPYING
Added LICENSE-tr2.txt, copied from the header of hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/tr2/optional.hpp
Added LICENSE-cJSON.txt, moved from the bottom of LICENSE.txt

* Generated license report for yarn-managed packages.
* Add LICENSE and NOTICES file of jaxb-api.
* Exclude LICENSE-binary-{yarn-applications-catalog-webapp|yarn-ui} from rat report.
These two files are autogenerated.
 2021-05-21 18:15:48 -07:00
Gautham B A
23ada04dac
HADOOP-17509. Parallelize building of dependencies (#2669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6c891c0dd4)
 2021-05-22 01:32:37 +08:00
Wei-Chiu Chuang
5e09d38b25
HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
 2021-05-22 00:31:59 +08:00