big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,297 Commits 2 Branches 2 Tags 264 MiB
5cc6f945da
Commit Graph

23297 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Shweta Yakkali
c561a70c49 HDDS-2213.Reduce key provider loading log level in OzoneFileSystem#getAdditionalTokenIssuers (#1556) 2019-10-11 10:23:19 -07:00
Szilard Nemeth
ec86f42e40 YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-11 14:01:19 +02:00
Szilard Nemeth
62b5cefaea YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal 2019-10-11 13:39:27 +02:00
Gabor Bota
4a700c20d5
HADOOP-16520. Race condition in DDB table init and waiting threads. (#1576). Contributed by Gabor Bota. 
Fixes HADOOP-16349. DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

Change-Id: Ia83e92b9039ccb780090c99c41b4f71ef7539d35
 2019-10-11 12:08:47 +02:00
Xiaoyu Yao
f267917ce3 HDDS-2282. scmcli pipeline list command throws NullPointerException. Contributed by Xiaoyu Yao. (#1642) 2019-10-10 22:33:59 -07:00
Bharat Viswanadham
9c72bf4621
HDDS-1986. Fix listkeys API. (#1588) 2019-10-10 16:49:06 -07:00
Eric Yang
31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN 
Contributed by Prabhu Joseph and Shane Kumpf
 2019-10-10 19:02:02 -04:00
lqjacklee
7a4b3d42c4
HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos. 
Contributed by lqjacklee.

Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c
 2019-10-10 21:58:42 +01:00
Bharat Viswanadham
957253fea6
HDDS-1984. Fix listBucket API. (#1555) 2019-10-10 12:33:55 -07:00
Bharat Viswanadham
4850b3aa86 HDDS-2269. Provide config for fair/non-fair for OM RW Lock. (#1623) 2019-10-10 23:28:19 +05:30
Steve Loughran
effe6087a5
HADOOP-16650. ITestS3AClosedFS failing. 
Contributed by Steve Loughran.

Change-Id: Ia9bb84bd6455e210a54cfe9eb944feeda8b58da9
 2019-10-10 17:32:25 +01:00
Ayush Saxena
104ccca916 HDFS-14900. Fix build failure of hadoop-hdfs-native-client. Contributed by Masatake Iwasaki. 2019-10-10 21:36:02 +05:30
Siddharth
a031388a2e HDDS-2266. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path. (#1633) 2019-10-10 15:30:11 +05:30
Ayush Saxena
eeb58a07e2 HDFS-14898. Use Relative URLS in Hadoop HDFS HTTP FS. Contributed by David Mollitor. 2019-10-10 01:03:33 +05:30
Surendra Singh Lilhore
d76e2655ac HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 2019-10-09 23:20:28 +05:30
Doroszlai, Attila
2d81abce5e
HDDS-2265. integration.sh may report false negative 
Closes #1608
 2019-10-09 16:17:40 +02:00
Márton Elek
b034350f3d
Squashed commit of the following: 
commit 9d6a1b97f365b22392ceac8d958cf041c3515bd6
Author: Doroszlai, Attila <adoroszlai@apache.org>
Date:   Mon Oct 7 13:42:35 2019 +0200

    HDDS-2265. integration.sh may report false negative
Revert "HDDS-2217. Remove log4j and audit configuration from the docker-config files"

This reverts commit 4b0a5bca46.
 2019-10-09 16:16:44 +02:00
Doroszlai, Attila
4b0a5bca46
HDDS-2217. Remove log4j and audit configuration from the docker-config files 
Closes #1582
 2019-10-09 15:51:00 +02:00
Chris Teoh
1f954e6798
HDDS-2217. Remove log4j and audit configuration from the docker-config files 
Closes #1582
 2019-10-09 15:46:16 +02:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Szilard Nemeth
35f093f5b3 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 2019-10-09 13:09:09 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Istvan Fajth
1d27930407 HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used (#1596) 2019-10-09 11:28:47 +05:30
Bharat Viswanadham
87d9f3668c
HDDS-2244. Use new ReadWrite lock in OzoneManager. (#1589) 2019-10-08 13:03:14 -07:00
Chen Liang
72ae371e7a HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-08 11:56:52 -07:00
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Siddharth
15a9beed1b HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS). (#1612) 2019-10-08 11:13:53 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Ayush Saxena
761594549e HDFS-14814. RBF: RouterQuotaUpdateService supports inherited rule. Contributed by Jinglun. 2019-10-08 14:01:44 +05:30
Anu Engineer
4fdf016358 HDDS-2245. Use dynamic ports for SCM in TestSecureOzoneCluster 
Contributed by kevin su.
 2019-10-07 15:41:42 -07:00
Doroszlai, Attila
012d897e5b HDDS-2262. SLEEP_SECONDS: command not found 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:38:54 -07:00
Doroszlai, Attila
aaa94c3da6 HDDS-2259. Container Data Scrubber computes wrong checksum 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:30:23 -07:00
Doroszlai, Attila
cfba6ac951 HDDS-2264. Improve output of TestOzoneContainer 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:10:57 -07:00
Doroszlai, Attila
1877312440 HDDS-2238. Container Data Scrubber spams log in empty cluster 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 13:59:49 -07:00
Surendra Singh Lilhore
382967be51 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-08 00:14:30 +05:30
Doroszlai, Attila
9685a6c0e5 HDDS-2239. Fix TestOzoneFsHAUrls (#1600) 2019-10-07 10:17:25 -07:00
bilaharith
1a77a15fe4
HADOOP-16587. Make ABFS AAD endpoints configurable. 
Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
 2019-10-07 13:07:46 +01:00
dchitlangia
7f332ebf8b
HDDS-2252. Enable gdpr robot test in daily build 
Closes #1602
 2019-10-07 11:35:39 +02:00
pingsutw
14cd969b6e
HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-10-07 16:38:08 +09:00
Shashikant Banerjee
022fe5f5b2 HDDS-2169. Avoid buffer copies while submitting client requests in Ratis. Contributed by Tsz-wo Sze(#1517). 2019-10-07 09:34:36 +05:30
Wei-Chiu Chuang
55c5436f39 Revert "HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár." 
This reverts commit 10bdc592d6.
 2019-10-05 11:16:11 -07:00
Steve Loughran
b8086bf54d
HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard. 
Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636
 2019-10-05 12:52:42 +01:00
Márton Elek
579dc87015
HDDS-2251. Add an option to customize unit.sh and integration.sh parameters 
Closes #1598
 2019-10-05 13:40:02 +02:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Vivek Ratnavel Subramanian
f209722a19 HDDS-2257. Fix checkstyle issues in ChecksumByteBuffer (#1603) 2019-10-04 16:36:32 -07:00
Doroszlai, Attila
a3cf54ccdc
HDDS-2250. Generated configs missing from ozone-filesystem-lib jars 
Closes #1597
 2019-10-04 23:31:22 +02:00
Hanisha Koneru
8de4374427
HDDS-2158. Fixing Json Injection Issue in JsonUtils. (#1486) 2019-10-04 12:52:29 -07:00
avijayanhwx
f3eaa84f9d HDDS-2164 : om.db.checkpoints is getting filling up fast. (#1536) 2019-10-04 12:44:21 -07:00
Norbert Kalmar
10bdc592d6 HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár. 2019-10-04 10:57:07 -07:00
Steve Loughran
6574f27fa3
HADOOP-16570. S3A committers encounter scale issues. 
Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks
of the S3A Committers.

* Thread pools are not cleaned up.
  This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,
  each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list
of files to load is the sole list which is passed around; .pendingset files are
loaded and processed in isolation -and reloaded if necessary for any
abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,
rather than that of individual pending commit files. The existing parallelized
Tasks API is still used to commit those files, but with a null thread pool, so
as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797
 2019-10-04 18:54:22 +01:00