big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,873 Commits 2 Branches 2 Tags 264 MiB
3228142f53
Commit Graph

5594 Commits

Author SHA1 Message Date
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
Ashutosh Gupta
21fa693d38
HADOOP-18238. Fix reentrancy check in SFTPFileSystem.close() (#4330) 
Contributed by Ashutosh Gupta
 2022-05-30 17:34:06 +01:00
HanleyYang
835f39cefc
HDFS-15878. RBF: Fix TestRouterWebHDFSContractCreate#testSyncable. (#4340). Contributed by Hanley Yang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:56:07 +05:30
Ritesh H Shukla
78008bc0ee
HADOOP-18245 Extend KMS related exceptions that get mapped to ConnectException (#4329) 2022-05-20 04:20:24 +08:00
slfan1989
f6fa5bd1aa
HADOOP-18229. Fix Hadoop-Common JavaDoc Errors (#4292) 
Contributed by slfan1989
 2022-05-18 12:12:04 +01:00
Steve Loughran
6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:54 +01:00
Lei Yang
6a95c3a039
HADOOP-18193:Support nested mount points in INodeTree 
Fixes #4181

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-11 17:01:21 -07:00
hchaverr
99a83fd4bd
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times 
Fixes #4266

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-10 13:58:39 -07:00
Ashutosh Gupta
40a8b9a6a5
HADOOP-17479. Fix the examples of hadoop config prefix (#4197) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:09:24 +09:00
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
liever18
2632d49282
HADOOP-18216. io.file.buffer.size must be positive (#4220) 
Documents that the buffer size must be > 0

Contributed by Jingxuan Fu.
 2022-04-26 19:53:11 +01:00
hchaverri
d60262fe00
HADOOP-18167. Add metrics to track delegation token secret manager op… (#4092) 
* HADOOP-18167. Add metrics to track delegation token secret manager operations
 2022-04-26 16:20:11 +00:00
Steve Loughran
1b4dba99b5
HADOOP-16202. Enhanced openFile(): hadoop-common changes. (#2584/1) 
This defines standard option and values for the
openFile() builder API for opening a file:

fs.option.openfile.read.policy
 A list of the desired read policy, in preferred order.
 standard values are
 adaptive, default, random, sequential, vector, whole-file

fs.option.openfile.length
 How long the file is.

fs.option.openfile.split.start
 start of a task's split

fs.option.openfile.split.end
 end of a task's split

These can be used by filesystem connectors to optimize their
reading of the source file, including but not limited to
* skipping existence/length probes when opening a file
* choosing a policy for prefetching/caching data

The hadoop shell commands which read files all declare "whole-file"
and "sequential", as appropriate.

Contributed by Steve Loughran.

Change-Id: Ia290f79ea7973ce8713d4f90f1315b24d7a23da1
 2022-04-24 17:33:04 +01:00
Ashutosh Gupta
f84b88dd6b
HADOOP-17564. Fix typo in UnixShellGuide.html (#4195) 
contributed by Ashutosh Gupta
 2022-04-22 17:59:41 +01:00
Renukaprasad C
4ff8a5dc73
HDFS-16526. Addendum Add metrics for slow DataNode (#4191) 2022-04-20 18:57:43 +05:30
Xing Lin
98b9c435f2
HADOOP-18172: Changed scope for isRootInternalDir/getRootFallbackLink for InodeTree (#4106) 
* HADOOP-18172: Change scope of InodeTree and its member methods to make them accessible from outside package.

Co-authored-by: Xing Lin <xinglin@linkedin.com>
 2022-04-19 22:47:02 -07:00
Renukaprasad C
f14f305051
HDFS-16526. Add metrics for slow DataNode (#4162) 2022-04-15 21:37:05 +05:30
caozhiqiang
cee8c62498
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) 2022-04-14 18:42:39 +09:00
GuoPhilipse
5de78ceb0e
HDFS-16516. Fix Fsshell wrong params (#4090). Contributed by GuoPhilipse. 2022-04-11 15:54:00 +08:00
Viraj Jasani
b69ede7154
HADOOP-18191. Log retry count while handling exceptions in RetryInvocationHandler (#4133) 2022-04-08 21:31:54 +09:00
Viraj Jasani
f70935522b
HADOOP-18188. Support touch command for directory (#4135) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-07 17:29:45 +09:00
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
Xing Lin
15a5ea2c95
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
 2022-03-31 21:59:09 +00:00
litao
ac50657c37
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) 2022-03-31 00:51:17 +09:00
zhongjingxiong
08e6d0ce60
HADOOP-18145. Fileutil's unzip method causes unzipped files to lose their original permissions (#4036) 
Contributed by jingxiong zhong
 2022-03-30 12:42:50 +01:00
Owen O'Malley
eb16421386 HDFS-16517 Distance metric is wrong for non-DN machines in 2.10. Fixed in HADOOP-16161, but 
this test case adds value to ensure the two getWeight methods stay in sync.

Fixes #4091

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 12:52:46 -07:00
Masatake Iwasaki
10876333ac Make upstream aware of 3.2.3 release. 2022-03-28 08:02:10 +00:00
PJ Fanning
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-26 20:31:16 +09:00
Steve Loughran
708a0ce21b
HADOOP-13704. Optimized S3A getContentSummary() 
Optimize the scan for s3 by performing a deep tree listing,
inferring directory counts from the paths returned.

Contributed by Ahmar Suhail.

Change-Id: I26ffa8c6f65fd11c68a88d6e2243b0eac6ffd024
 2022-03-22 13:21:12 +00:00
Owen O'Malley
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 09:27:35 -07:00
Abhishek Das
da9970dd69 HADOOP-18129: Change URI to String in INodeLink to reduce memory footprint of ViewFileSystem 
Fixes #3996
 2022-03-17 17:25:55 -07:00
Steve Loughran
9037f9a334
HADOOP-18162. hadoop-common support for MAPREDUCE-7341 Manifest Committer 
* New statistic names in StoreStatisticNames
  (for joint use with s3a committers)
* Improvements to IOStatistics implementation classes
* RateLimiting wrapper to guava RateLimiter
* S3A committer Tasks moved over as TaskPool and
  added support for RemoteIterator
* JsonSerialization.load() to fail fast if source does not exist

+ tests.

This commit is a prerequisite for the main MAPREDUCE-7341 Manifest Committer
patch.

Contributed by Steve Loughran

Change-Id: Ia92e2ab5083ac3d8d3d713a4d9cb3e9e0278f654
 2022-03-17 11:20:53 +00:00
Xing Lin
8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 11:29:48 -07:00
Owen O'Malley
7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
Mukund Thakur
672e380c4f
HADOOP-18112: Implement paging during multi object delete. (#4045) 
Multi object delete of size more than 1000 is not supported by S3 and 
fails with MalformedXML error. So implementing paging of requests to 
reduce the number of keys in a single request. Page size can be configured
using "fs.s3a.bulk.delete.page.size" 

 Contributed By: Mukund Thakur
 2022-03-11 13:05:45 +05:30
Gautham B A
d0fa9b5775
HADOOP-18155. Refactor tests in TestFileUtil (#4053) 2022-03-10 22:02:38 +05:30
Duo Zhang
db36747e83
HADOOP-17526 Use Slf4jRequestLog for HttpRequestLog (#4050) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-10 10:15:09 +08:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Viraj Jasani
278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 2022-03-04 22:17:48 +05:30
Chao Sun
f800b65b40 Make upstream aware of 3.3.2 release 2022-03-02 19:14:50 -08:00
ted12138
902a7935e9
HADOOP-18128. Fix typo issues of outputstream.md (#4025) 2022-03-02 18:25:56 +08:00
Ayush Saxena
d05655d2ad
Revert "HADOOP-18082.Add debug log when RPC#Reader gets a Call. (#3891). Contributed by JiangHua Zhu." 
Exposes a Race Condition. Which leads to test failures in YARN. (HADOOP-18143)

This reverts commit 2025243fbf.
 2022-02-28 21:44:24 +05:30
Owen O'Malley
12fa38d546
HADOOP-18139: Allow configuration of zookeeper server principal. 
Fixes #4024

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-02-24 15:01:50 -08:00
monthonk
1f157f802d
HADOOP-17386. Change default fs.s3a.buffer.dir to be under Yarn container path on yarn applications (#3908) 
Co-authored-by: Monthon Klongklaew <monthonk@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-22 13:50:27 +09:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Steve Loughran
cae749b076
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I73af19d2e2e41f4ba686c470726a80c3903a1950
 2022-02-21 17:08:56 +00:00
jianghuazhu
2025243fbf
HADOOP-18082.Add debug log when RPC#Reader gets a Call. (#3891). Contributed by JiangHua Zhu. 2022-02-17 01:49:45 +05:30
Chentao Yu
19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
GuoPhilipse
b68964336d
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-14 05:38:28 +09:00