big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,891 Commits 2 Branches 2 Tags
Commit Graph

24891 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
litao
9132eeb4dd HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc48023c0c71dd428df40a4352feede373b)
 2022-04-23 12:15:52 +09:00
Ashutosh Gupta
261b11f7db HADOOP-17564. Fix typo in UnixShellGuide.html (#4195) 
contributed by Ashutosh Gupta
 2022-04-22 18:00:58 +01:00
Boyina, Hemanth Kumar
30061940af
HADOOP-17588. CryptoInputStream#close() should be synchronized. 
Contributed by RenukaPrasad C

Change-Id: Ia0a19ccc55a67e5434f0be23a500496bc7682f40
 2022-04-21 22:11:37 +01:00
Giovambattista Vieri
c4de4add5b
HADOOP-18214. Update BUILDING.txt (#3811) 
java-8-openjdk become openjdk-8-jdk (see both ubuntu and debian package's name)

Contributed by Giovambattista Vieri
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>

Change-Id: I63b2bbfdd575cf56d20cd6c8fff33a70cadda7f2
 2022-04-21 18:39:51 +01:00
Ashutosh Gupta
2526dbf428
HADOOP-17551. Upgrade maven-site-plugin to 3.11.0 (#4196) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 56cfd6061770872ce35cf815544b0c0f49613209)
 2022-04-21 22:16:58 +09:00
S O'Donnell
5e137ac33e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 8ae033d1a3d65281d3ad1f07f430e74d3a01b1b9.
 2022-04-20 20:45:17 +01:00
Xing Lin
7ae328949e HADOOP-18172: Changed scope for isRootInternalDir/getRootFallbackLink for InodeTree (#4106) 
Co-authored-by: Xing Lin <xinglin@linkedin.com>
(cherry picked from commit 98b9c435f24c76e3629f6d8e9c14473b46fe6a0e)
 2022-04-20 10:55:48 -07:00
Steve Loughran
caecec45f5
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939) 
Reviewed-by: Siyao Meng <siyao@apache.org>

Contributed by Viraj Jasani
 2022-04-20 16:36:51 +01:00
Dongjoon Hyun
af3558d61a
HADOOP-17341. Upgrade commons-codec to 1.15 (#2428) 
Change-Id: Iab26db901570b507ab25ddbf316a9579a9e92620
Reviewed-by: Chao Sun <sunchao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-20 12:29:00 +01:00
qinyuren
2ff91232bc HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd173749f2af4f17946fb37c4c72e2de26764)
 2022-04-20 15:07:38 +09:00
Stephen O'Donnell
8ae033d1a3 HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee03639f41a022dd07d5fc04e3aa65a94b5f)
 2022-04-19 11:12:36 +01:00
qinyuren
c913dc3072 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0d5f522f3b799ea72c6c887d5d2c42d)
 2022-04-19 13:38:58 +09:00
jianghuazhu
cfe2d8aa79
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e7120e7af250ebf0b4bafb40b412d3e11c)
 2022-04-19 11:13:40 +08:00
Steve Loughran
51a532a532
HADOOP-18202. create-release fails fatal: unsafe repository (#4188) 
Since April 2022/CVE-2022-24765, git refuses to work in directories
whose owner != the current user, unless explicitly told to trust it.

This patches the create-release script to trust the /build/source
dir mounted from the hosting OS, whose userid is inevitably different
from that of the account in the container running git.

Contributed by: Steve Loughran, Ayush Saxena and the new git error messages

Change-Id: I855a105e6d0ab533468f9436578c8d4f81b0840b
 2022-04-18 19:28:44 +01:00
Ashutosh Gupta
1eb4f9ef04
HDFS-16536. TestOfflineImageViewer fails on branch-3.3 (#4182) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-18 16:05:42 +09:00
Quanlong Huang
9ae903dd1b
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
(cherry picked from commit 35d4c02bccd303b5a2608a952fa1f8487001087b)
 2022-04-18 10:35:08 +08:00
Daniel Carl Jones
eab586d566
HADOOP-18201. Remove endpoint config overrides for ITestS3ARequesterPays (#4169) 
Contributed by Daniel Carl Jones.

Change-Id: Icf99cc878e2b0ba92df630a8aa578616f96e09cc
 2022-04-14 16:47:24 +01:00
daimin
0ef1a13f01
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 
(cherry picked from commit c65c383b7ebef48c638607f15ba35d61554982cb)
 2022-04-14 16:16:36 +08:00
Takanobu Asanuma
52abc9f132 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959ca0a68c52ed6a9c721704e57afbcc7)
 2022-04-14 11:50:42 +09:00
Steve Loughran
52c6d77274
HDFS-14478. Add libhdfs APIs for openFile (#4166) 
Contributed by Sahil Takiar

Change-Id: I2f9e82a69010df7496704754515b031f2a907167
 2022-04-13 16:01:53 +01:00
Akira Ajisaka
2112ef61e0
YARN-10553. Refactor TestDistributedShell (#4159) 
(cherry picked from commit 890f2da624465473a5f401a3bcfc4bbd068289a1)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSWithMultipleNodeManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java

Co-authored-by: Ahmed Hussein <50450311+amahussein@users.noreply.github.com>
 2022-04-13 12:22:17 +09:00
qinyuren
07dface36a HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a112334e48087bd60674538af739922a)
 2022-04-13 11:48:19 +09:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
Viraj Jasani
e5516cdfaf HADOOP-18191. Log retry count while handling exceptions in RetryInvocationHandler (#4133) 
(cherry picked from commit b69ede7154d44538a4a66824c34f7ba143deef25)
 2022-04-11 15:23:55 +09:00
Hanisha Koneru
9da7d80c4e HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 
(cherry picked from commit e62d8f841275ee47a0ba911415aac9e39af291c6)
 2022-04-11 15:19:18 +09:00
singer-bin
26705bbc60
HDFS-16457. Make fs.getspaceused.classname reconfigurable (apache#4069) (#4156) 2022-04-11 14:59:34 +09:00
Akira Ajisaka
603367c54f
HADOOP-18178. Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2 (#4147) 
(cherry picked from commit 4b786c797a76d00ef33237fde76f2bd6edda7f3a)

 Conflicts:
	LICENSE-binary

Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-11 14:58:28 +09:00
Takanobu Asanuma
30afe7ca20 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 
(cherry picked from commit 37650ced81964efe7a9ea45bf505af96b8035d53)
 2022-04-08 18:39:26 +09:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00
Stephen O'Donnell
bd0dbf319a HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf76ce661e0b10bb5d16cb419f9b9b10)
 2022-04-06 11:54:58 +01:00
Viraj Jasani
b2eee14f2e HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7c20602b17725cf266075477d70219a2a03721aa)
 2022-04-06 18:55:09 +09:00
Viraj Jasani
a6fb77f7eb HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 
(cherry picked from commit 278568203b9c2033743ecca60dbc62d397a85a8d)
 2022-04-06 18:40:07 +09:00
wangzhaohui
0e621c890d HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a73a18cf05aeb2c40429c34c4b9b0732)
 2022-04-06 17:30:39 +09:00
Xing Lin
20483f6dc7
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-31 15:17:26 -07:00
Xing Lin
ecafd38c09
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. (#4123) 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02df18424b2406fd66b34c1bdf3d7ab55)

Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-03-31 20:26:09 +00:00
litao
0ecb34f8f6
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) (#4125) 2022-03-31 19:24:05 +09:00
litao
cfca024190 HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:03:27 -07:00
Owen O'Malley
e24bd1c15b HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Fixed in HADOOP-16161, but 
this test case adds value to ensure the two getWeight methods stay in sync.

Fixes #4091

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-30 14:58:03 -07:00
Kengo Seki
85843f2158 MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (#4120) 
(cherry picked from commit dc4a680da8bcacf152cc8638d86dd171a7901245)
 2022-03-30 13:49:45 +00:00
zhongjingxiong
1ee93f7947
HADOOP-18145. Fileutil's unzip method causes unzipped files to lose their original permissions (#4036) 
Contributed by jingxiong zhong

Change-Id: Ie252e609798719dc658364f9bae48b34dc72a79c
 2022-03-30 12:52:52 +01:00
Lei Yang
4f85c9a73b HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:28:38 -07:00
Masatake Iwasaki
f8314cd469 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a2449ec49f840d93e1c7d290c5218ef4ea)
 2022-03-28 08:08:59 +00:00
Masatake Iwasaki
419d9718a8 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 10876333acd4194da3cb545c4f7a2b2cc88be959)
 2022-03-28 08:03:39 +00:00
GuoPhilipse
6300fc1fcd
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4a895b98ccd41dde1c9524a6bb0ea31)
 2022-03-27 22:24:44 +09:00
Peter Bacsko
ab5ea272ec YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 
(cherry picked from commit a0deda1a777d8967fb8c08ac976543cda895773d)
 2022-03-24 14:11:23 +09:00
Daniel Carl Jones
b749438a8c
HADOOP-14661. Add S3 requester pays bucket support to S3A (#3962) 
Adds the option fs.s3a.requester.pays.enabled, which, if set to true, allows
the client to access S3 buckets where the requester is billed for the IO.

Contributed by Daniel Carl Jones

Change-Id: I51f64d0f9b3be3c4ec493bcf91927fca3b20407a
 2022-03-23 20:13:20 +00:00
Steve Loughran
ebdd8771c5
HADOOP-17851. S3A to support user-specified content encoding (#3498) 
The option fs.s3a.object.content.encoding declares the content encoding to be set on files when they are written; this is served up in the "Content-Encoding" HTTP header when reading objects back in.

This is useful for people loading the data into other tools in the AWS ecosystem which don't use file extensions to infer compression type (e.g. serving compressed files from S3 or importing into RDS)

Contributed by: Holden Karau

Change-Id: Ice0da75b516370f51f79e45f391d46c5c7aa4ce4
 2022-03-23 20:05:28 +00:00
qinyuren
55f4baf3e4 HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50c3ff65676d946e96bbc7846ad3131a)
 2022-03-23 15:05:24 +09:00
André Fonseca
5e102a6ebb
HADOOP-18160 Avoid shading wildfly.openssl runtime dependency (#4074) 
Contributed by André Fonseca.

Change-Id: I198c66b58c0e05c1752ad9d6a50fe2f2932f61d5
 2022-03-22 18:51:48 +00:00
Steve Loughran
105e0dbd92
HADOOP-13704. Optimized S3A getContentSummary() 
Optimize the scan for s3 by performing a deep tree listing,
inferring directory counts from the paths returned.

Contributed by Ahmar Suhail.

Change-Id: I26ffa8c6f65fd11c68a88d6e2243b0eac6ffd024
 2022-03-22 13:31:13 +00:00