big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,535 Commits 2 Branches 2 Tags 264 MiB
a1f90883f0
Commit Graph

24535 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
a1f90883f0
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d)
 2021-07-12 10:18:19 +09:00
sumangala-patki
aa6a9cac72
HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#3106) 
* HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795)
. Contributed by Sumangala Patki.

(cherry picked from commit 76d92eb2a2)
 2021-07-10 15:09:59 +05:30
litao
7cb91db575 HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-10 13:56:36 +05:30
Artem Smotrakov
c18ad9a007 HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)
 2021-07-10 13:43:31 +09:00
Akira Ajisaka
6cc9549499 HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 
(cherry picked from commit 6ac5d8a1ed)
 2021-07-09 22:50:18 +09:00
S O'Donnell
705884b87a HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76)
 2021-07-09 10:19:47 +01:00
bshashikant
ab939814a3 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9a)
 2021-07-09 09:17:10 +01:00
Rafal Wojdyla
e3fb63f33f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-07 22:43:31 +01:00
Eric Payne
e395711164 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 
(cherry picked from commit 7581413156)
 2021-07-07 20:57:32 +00:00
Masatake Iwasaki
99cff01e38
HADOOP-17775. Remove JavaScript package from Docker environment. (#3182) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 17:49:19 +09:00
liangxs
24b780820c HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc)
 2021-07-07 09:41:11 +08:00
Mukund Thakur
e8f9af6f2a
HADOOP-17250 Lot of short reads can be merged with readahead. (#3110) 
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur

Change-Id: Ib920eedd0087caa150afa4d4c23e89df56b29e83
 2021-07-05 11:23:32 +01:00
Viraj Jasani
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf)
 2021-07-05 13:42:51 +09:00
litao
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e4)
 2021-07-05 08:15:10 +09:00
Mehakmeet Singh
f1a14df9e6
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value (#3144) 
Contributed by: Mehakmeet Singh

Change-Id: I3302654ca36474a5f399aa848f88bce4587022d8
 2021-07-02 14:13:26 +01:00
Viraj Jasani
bf6ed21104 HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:59:25 -07:00
Wanqiang Ji
628b08db63
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 
(cherry picked from commit c005195b78)
 2021-06-30 09:35:36 +08:00
litao
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 
(cherry picked from commit 95454d821c)
 2021-06-29 18:15:56 +08:00
Viraj Jasani
8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073)
 2021-06-29 18:13:00 +08:00
Jim Brennan
47b3939009 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 
(cherry picked from commit dc6f456e95)
 2021-06-28 14:55:26 +00:00
Jim Brennan
75f8198aa8 YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 
(cherry picked from commit 7c7d02edbd)
 2021-06-25 20:36:41 +00:00
Zamil Majdy
80859d714d
HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt (#3109) 
Contributed by Zamil Majdy.

Change-Id: I680d9c425c920ff1a7cd4764d62e10e6ac78bee4
 2021-06-25 20:47:11 +01:00
Ahmed Hussein
22e7567475
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3130). Contributed by Ahmed Hussein. (#3138) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 581f43dce1)
 2021-06-25 22:48:56 +05:30
Steve Loughran
39e6f2d191
HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain." (#3133) 
This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint
is set in fs.s3a.endpoint, S3A filesystem creation may fail on
non-EC2 deployments, depending on the local host environment setup.

* If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region
  is null, the region is set to "us-east-1".
* If fs.s3a.endpoint.region is explicitly set to "" then the client
  falls back to the SDK region resolution chain; this works on EC2
* Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+
* Also contains some minor restructuring of troubleshooting.md

Contributed by Steve Loughran.

Change-Id: Ife482cff513307cd52d59eec56beac0a33e031f5
 2021-06-24 16:38:55 +01:00
litao
4ca49fcd44 HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 16:00:47 +05:30
litao
5fc61d2e56 HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 
(cherry picked from commit fdef2b4cca)
 2021-06-24 13:27:33 +09:00
Masatake Iwasaki
b672edb64c YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 
(cherry picked from commit 11d144d228)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock
 2021-06-23 10:50:59 +00:00
Viraj Jasani
68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 
(cherry picked from commit d9fbb3c508)
 2021-06-23 11:24:00 +08:00
Viraj Jasani
4825c7c28d MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa)
 2021-06-21 16:47:37 +09:00
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
 2021-06-21 10:28:05 +09:00
litao
7d202e8365 HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:24:03 +05:30
Viraj Jasani
ef3a2a3b3b HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:52 +05:30
Ahmed Hussein
92ade1f6f9 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:49 +05:30
litao
8d241482bd HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:38 +05:30
Hui Fei
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3113) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 14:41:02 +08:00
Wei-Chiu Chuang
577b96ef42
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548)
 2021-06-17 11:39:34 +08:00
Takanobu Asanuma
25138c98bf HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129)
 2021-06-17 10:00:36 +09:00
Ayush Saxena
3e03ecd680 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:54:36 +05:30
Jim Brennan
57ee4238eb YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit 9a6a11c452)
 2021-06-15 19:30:16 +00:00
Takanobu Asanuma
546c66b812 HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682)
 2021-06-15 17:22:34 +09:00
Ahmed Hussein
840612b3b1 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 
(cherry picked from commit 747883af4d)
 2021-06-14 14:37:27 +00:00
Takanobu Asanuma
10000f162d HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. (#3090) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580)
 2021-06-14 10:21:30 +09:00
Petre Bogdan Stolojan
254d943126
HADOOP-17547 Magic committer to downgrade abort in cleanup if list uploads fails with access denied (#3051) 
Contributed by Bogdan Stolojan

Change-Id: I32d6dc4f72087783a3ea12473d11690ac14fe3cb
 2021-06-12 17:46:11 +01:00
litao
52d055a7eb HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0)
 2021-06-11 17:33:06 +09:00
Mingliang Liu
91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 8c0f948054.
 2021-06-11 00:35:41 -07:00
Akira Ajisaka
19c969981d
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. (#3068) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9a17f97133)
 2021-06-10 14:33:41 +09:00
Viraj Jasani
0d6939ffc4
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca27744770)
 2021-06-09 10:38:38 +08:00
Wei-Chiu Chuang
83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. (#3054) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-08 19:32:34 -07:00
Viraj Jasani
8f0ba9ee1b
HADOOP-17725. Improve error message for token providers in ABFS (#3041) 
Contributed by Viraj Jasani.
 2021-06-08 22:05:01 +01:00
Steve Loughran
4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.

Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
 2021-06-08 22:05:00 +01:00