big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,681 Commits 2 Branches 2 Tags 264 MiB
c4e8826194
Commit Graph

24681 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jim Brennan
392a62278a YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger 
(cherry picked from commit 4a7deae478)
 2020-08-17 16:55:35 +00:00
Steve Loughran
49f8ae965e
HADOOP-13230. S3A to optionally retain directory markers. 
This adds an option to disable "empty directory" marker deletion,
so avoid throttling and other scale problems.

This feature is *not* backwards compatible.
Consult the documentation and use with care.

Contributed by Steve Loughran.

Change-Id: I69a61e7584dc36e485d5e39ff25b1e3e559a1958
 2020-08-15 20:19:49 +01:00
hemanthboyina
5ededc0cba YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T. 
(cherry picked from commit 86bbd38c8d)
 2020-08-13 23:47:50 +05:30
Mukund Thakur
571737f4ac HADOOP-17192. ITestS3AHugeFilesSSECDiskBlock failing (#2221) 
Contributed by Mukund Thakur
 2020-08-13 14:33:27 +01:00
Uma Maheswara Rao G
99b120a06e HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test
 2020-08-13 14:32:19 +01:00
S O'Donnell
42154f04e3 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 
(cherry picked from commit 10716040a8)
 2020-08-12 09:33:14 +01:00
Akira Ajisaka
1354400e7c
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584b)
 2020-08-12 16:09:15 +09:00
Gautham B A
b4a105a209
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3)
 2020-08-11 16:35:41 +09:00
Jonathan Hung
706d14c41d YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 64753addba)
 2020-08-07 17:43:16 -07:00
Ayush Saxena
2943e6650f HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:20:42 +05:30
bibinchundatt
0c0ea4979e YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 
(cherry picked from commit 81da221c75)
 2020-08-07 09:01:53 +05:30
Wanqiang Ji
ffb96775e3
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86)
 2020-08-06 09:36:34 +09:00
Eric Badger
723f1f61d6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925e)
 2020-08-05 18:56:56 +00:00
Mingliang Liu
1cd1f978da
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 20:49:50 -07:00
Mingliang Liu
7b41d4ac74
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f7)
 2020-08-05 10:51:38 +09:00
sguggilam
97dd1cb57e
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:31:26 -07:00
Mukund Thakur
251d2d1fa5
HADOOP-17131. Refactor S3A Listing code for better isolation. (#2148) 
Contributed by Mukund Thakur.

Change-Id: I79160b236a92fdd67565a4b4974f1862e600c210
 2020-08-04 17:13:06 +01:00
hemanthboyina
b5d712251c HDFS-15503. File and directory permissions are not able to be modified from WebUI. 
(cherry picked from commit 82f3ffcd64)
 2020-08-03 23:43:29 +05:30
Brahma Reddy Battula
643ff4881d YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 
(cherry picked from commit eac558380f)
 2020-08-03 12:55:31 +05:30
Uma Maheswara Rao G
4fe491d10e HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:31:37 -07:00
Uma Maheswara Rao G
ae8261c671 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:31:15 -07:00
Uma Maheswara Rao G
10f8010519 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:30:54 -07:00
Uma Maheswara Rao G
aea1a8e2bd HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:30:30 -07:00
Uma Maheswara Rao G
35fe6fd54f HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5)
 2020-07-31 01:29:46 -07:00
Yuan
e1ac16a318
HDFS-14950. fix missing libhdfspp lib in dist-package (#1947) 
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit e756fe3590)
 2020-07-31 16:51:05 +09:00
Eric E Payne
0cb1b178ec YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166)
 2020-07-30 16:39:56 +00:00
ywheel
2efa28cb79
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 
(cherry picked from commit cf4eb75608)
 2020-07-30 13:28:35 +09:00
Jonathan Hung
d3c7f7cdbd YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 
(cherry picked from commit 3eaf62726f)
 2020-07-28 13:43:44 -07:00
Akira Ajisaka
9263b75b3c
YARN-10367. Failed to get nodejs 10.21.0 when building docker image (#2171) 
(cherry picked from commit e277d338da)
 2020-07-27 00:53:24 +09:00
Sneha Vijayarajan
18ca80331c
Hadoop 17132. ABFS: Fix Rename and Delete Idempotency check trigger 
- Contributed by Sneha Vijayarajan
 2020-07-25 13:13:18 +00:00
ishaniahuja
f24e2ec487
HADOOP-17058. ABFS: Support for AppendBlob in Hadoop ABFS Driver 
- Contributed by Ishani Ahuja
 2020-07-25 13:12:32 +00:00
Mehakmeet Singh
7c9b459786
HADOOP-16961. ABFS: Adding metrics to AbfsInputStream (#2076) 
Contributed by Mehakmeet Singh.
 2020-07-25 13:12:09 +00:00
Mehakmeet Singh
bbd3278d09
HADOOP-17065. Add Network Counters to ABFS (#2056) 
Contributed by Mehakmeet Singh.
 2020-07-25 13:11:34 +00:00
Karthik Amarnath
8b7e77443d
HDFS-15168: ABFS enhancement to translate AAD to Linux identities. (#1978) 2020-07-25 13:10:39 +00:00
Sneha Vijayarajan
903935da0f
HADOOP-17053. ABFS: Fix Account-specific OAuth config setting parsing 
Contributed by Sneha Vijayarajan
 2020-07-25 13:10:30 +00:00
Sneha Vijayarajan
869a68b81e
HADOOP-16852: Report read-ahead error back 
Contributed by Sneha Vijayarajan
 2020-07-25 13:10:19 +00:00
Sneha Vijayarajan
27b20f9689
HADOOP-17054. ABFS: Fix test AbfsClient authentication instance 
Contributed by Sneha Vijayarajan
 2020-07-25 13:09:26 +00:00
Sneha Vijayarajan
eed06b46eb
Hadoop-17015. ABFS: Handling Rename and Delete idempotency 
Contributed by Sneha Vijayarajan.
 2020-07-25 13:08:01 +00:00
Eric Badger
7ec692aa83 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
 2020-07-24 22:46:57 +00:00
bilaharith
1ae72d2438
HADOOP-17092. ABFS: Making AzureADAuthenticator.getToken() throw HttpException 
- Contributed by Bilahari T H

Change-Id: Id9576d9509faaf057bf419ccb1879ac0cef7a07b
 2020-07-22 18:26:36 +01:00
Akira Ajisaka
d35f7eaeb9
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 
(cherry picked from commit d5b4766158)
 2020-07-23 00:40:31 +09:00
Ayush Saxena
e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Chen Liang
c8c40be761 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 15:22:34 -07:00
Ayush Saxena
8a58a12626 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 2020-07-20 22:26:11 +05:30
Steve Loughran
5aa9396a58
HADOOP-17107. hadoop-azure parallel tests not working on recent JDKs (#2118) 
Contributed by Steve Loughran.

Change-Id: I972264aed36f384b7ae23e214326ef7870261cf5
 2020-07-20 10:54:22 +01:00
bilaharith
e01852181a
HADOOP-16682. ABFS: Removing unnecessary toString() invocations 
- Contributed by Bilahari T H

Change-Id: Id55495b44d81533d1d3654de2553c709f505f7eb
 2020-07-20 10:53:59 +01:00
Mukund Thakur
187b161c20
HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: Ic0c9c44a1b14b4bfa962cede3f782a98fdfd7513
 2020-07-16 18:15:40 +01:00
Mehakmeet Singh
0d88ed2794
HADOOP-17129. Validating storage keys in ABFS correctly (#2141) 
Contributed by Mehakmeet Singh

Change-Id: I8016ee2f9ffbc86ea867f4a3d960b134e507d099
 2020-07-16 18:11:52 +01:00
Ahmed Hussein
9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Erik Krogen
67e01ed2ca HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 
(cherry picked from 317fe4584a)
 2020-07-15 08:26:38 -07:00