big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,334 Commits 2 Branches 2 Tags 264 MiB
4890b855ec
Commit Graph

24334 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same 
Contributed by Gabor Bota.
 2020-10-13 16:17:44 +01:00
Mukund Thakur
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:31:00 +01:00
Steve Loughran
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380)" 
(reverting to change the JIRA number)

This reverts commit 3e1b1dfd77.

Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd
 2020-10-13 13:30:38 +01:00
Mukund Thakur
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.
 2020-10-13 13:27:55 +01:00
Pranav Bheda
be3edd0532
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 
* update org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.13
* update org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13
 2020-10-13 17:54:41 +09:00
Szilard Nemeth
42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378). Contributed by Uma Maheswara Rao G. 2020-10-12 14:22:44 -07:00
Swaroopa Kadam
2e46ef9417
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 10:57:02 -07:00
Dongjoon Hyun
b92f72758b
HADOOP-17258. Magic S3Guard Committer to overwrite existing pendingSet file on task commit (#2371) 
Contributed by Dongjoon Hyun and Steve Loughran

Change-Id: Ibaf8082e60eff5298ff4e6513edc386c5bae0274
 2020-10-12 13:39:15 +01:00
Akira Ajisaka
69ef9b1ee8
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 09:52:12 +09:00
LeonGao91
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 2020-10-10 22:08:36 +05:30
Akira Ajisaka
2de7240d4f
HADOOP-17269. [JDK 11] Upgrade SpotBugs to 4.1.3 to fix false-positive warnings. (#2374) 2020-10-10 00:28:05 +09:00
Hui Fei
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 2020-10-09 16:12:22 +08:00
Jim Brennan
deb35a32ba YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:46:47 +00:00
Steve Loughran
f83e07a20f HADOOP-17293. S3A to always probe S3 in S3A getFileStatus on non-auth paths 
This reverts changes in HADOOP-13230 to use S3Guard TTL in choosing when
to issue a HEAD request; fixing tests to compensate.

New org.apache.hadoop.fs.s3a.performance.OperationCost cost,
S3GUARD_NONAUTH_FILE_STATUS_PROBE for use in cost tests.

Contributed by Steve Loughran.

Change-Id: I418d55d2d2562a48b2a14ec7dee369db49b4e29e
 2020-10-08 15:35:57 +01:00
Jinglun
52db86b0bb
HADOOP-17021. Add concat fs command (#1993) 
Contributed by Jinglun
 2020-10-08 10:36:07 +01:00
Karthik Palanisamy
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 2020-10-08 12:51:04 +05:30
Karthik Palanisamy
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 2020-10-07 22:09:04 +05:30
Mukund Thakur
82522d60fb
HADOOP-17281 Implement FileSystem.listStatusIterator() in S3AFileSystem (#2354) 
Contains HADOOP-17300: FileSystem.DirListingIterator.next() call should 
return NoSuchElementException

Contributed by Mukund Thakur
 2020-10-07 13:59:06 +01:00
bshashikant
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. (#2355) 2020-10-07 10:34:32 +05:30
Akira Ajisaka
074f0d46af
HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360) 
* Support getHAServiceState in DFSRouter
 2020-10-07 13:37:45 +09:00
hemanthboyina
921ca1f554 HDFS-15543. RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled. Contributed by Hemanth Boyina. 2020-10-07 09:58:53 +05:30
Doroszlai, Attila
1cfe5916e2 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-06 14:58:42 -07:00
Jim Brennan
b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Liang-Chi Hsieh
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 
This switches the SnappyCodec to use the java-snappy codec, rather than the native one. 

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh
 2020-10-06 17:07:54 +01:00
Ikko Ashimine
4347a5c955
HADOOP-17294. Fix typos existance to existence (#2357) 2020-10-06 10:10:44 +09:00
Fei Hui
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356). Contributed by Hui Fei. 2020-10-05 22:11:24 +05:30
Siyao Meng
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir (#2352) 2020-10-05 05:02:00 -07:00
Akira Ajisaka
7d4bcb312b
HADOOP-17297. Use Yetus before YETUS-994 to enable adding comments to GitHub (#2359) 2020-10-05 19:16:37 +09:00
Adam Antal
a1f7e760df YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-05 10:09:14 +02:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Arpit Agarwal
18fa4397e6
MAPREDUCE-7298. Distcp doesn't close the job after the job is completed. Contributed by Aasha Medhi. 
Change-Id: I63d249bbb18ccedaeee9f10123a78e32f9e54ed2
 2020-10-02 08:29:55 -07:00
bilaharith
51598d8b1b
HADOOP-17183. ABFS: Enabling checkaccess on ABFS (#2331) 
Contributed by Bilahari TH
 2020-10-01 21:29:05 +01:00
Adam Antal
bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
maobaolong
d68d2a5c1e
HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:22:55 -07:00
He Xiaoqiao
a490d87eb7 HADOOP-17280. Service-user cost shouldn't be accumulated to totalDecayedCallCost and totalRawCallCost. Contributed by Jinglun. 2020-09-30 12:12:52 +08:00
Xiaoyu Yao
4c5ad57818
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334) 2020-09-29 20:15:27 -07:00
huangtianhua
a89ca56a1b
HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347) 2020-09-29 19:00:55 -07:00
Szilard Nemeth
a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Karen Coppage
75d10f8499
HADOOP-17267. Add debug-level logs in Filesystem.close() (#2321) 
When a filesystem is closed, the FileSystem log will, at debug level,
log the method calling close/closeAll.

At trace level: the full calling stack.

Contributed by Karen Coppage.
 2020-09-29 16:06:50 +01:00
Wanqiang Ji
143bdd4188
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 2020-09-29 23:06:25 +09:00
Ayush Saxena
d686a655bc HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui. 2020-09-29 00:03:11 +05:30
Ayush Saxena
fc8a6dd8a9 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 2020-09-28 23:55:31 +05:30
Fei Hui
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345) 2020-09-28 09:54:01 -07:00
Inigo Goiri
eae0035a2d HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao. 2020-09-28 09:40:39 -07:00
Takanobu Asanuma
d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Aryan Gupta
bbbfa7d415
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 2020-09-27 03:18:01 +09:00
Uma Maheswara Rao G
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00