big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,831 Commits 2 Branches 2 Tags 264 MiB
3dc9db3aed
Commit Graph

23831 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Márton Elek
b034350f3d
Squashed commit of the following: 
commit 9d6a1b97f365b22392ceac8d958cf041c3515bd6
Author: Doroszlai, Attila <adoroszlai@apache.org>
Date:   Mon Oct 7 13:42:35 2019 +0200

    HDDS-2265. integration.sh may report false negative
Revert "HDDS-2217. Remove log4j and audit configuration from the docker-config files"

This reverts commit 4b0a5bca46.
 2019-10-09 16:16:44 +02:00
Doroszlai, Attila
4b0a5bca46
HDDS-2217. Remove log4j and audit configuration from the docker-config files 
Closes #1582
 2019-10-09 15:51:00 +02:00
Chris Teoh
1f954e6798
HDDS-2217. Remove log4j and audit configuration from the docker-config files 
Closes #1582
 2019-10-09 15:46:16 +02:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Szilard Nemeth
35f093f5b3 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 2019-10-09 13:09:09 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Istvan Fajth
1d27930407 HDDS-2233 - Remove ByteStringHelper and refactor the code to the place where it used (#1596) 2019-10-09 11:28:47 +05:30
Bharat Viswanadham
87d9f3668c
HDDS-2244. Use new ReadWrite lock in OzoneManager. (#1589) 2019-10-08 13:03:14 -07:00
Chen Liang
72ae371e7a HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-08 11:56:52 -07:00
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Siddharth
15a9beed1b HDDS-2260. Avoid evaluation of LOG.trace and LOG.debug statement in the read/write path (HDDS). (#1612) 2019-10-08 11:13:53 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Ayush Saxena
761594549e HDFS-14814. RBF: RouterQuotaUpdateService supports inherited rule. Contributed by Jinglun. 2019-10-08 14:01:44 +05:30
Anu Engineer
4fdf016358 HDDS-2245. Use dynamic ports for SCM in TestSecureOzoneCluster 
Contributed by kevin su.
 2019-10-07 15:41:42 -07:00
Doroszlai, Attila
012d897e5b HDDS-2262. SLEEP_SECONDS: command not found 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:38:54 -07:00
Doroszlai, Attila
aaa94c3da6 HDDS-2259. Container Data Scrubber computes wrong checksum 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:30:23 -07:00
Doroszlai, Attila
cfba6ac951 HDDS-2264. Improve output of TestOzoneContainer 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 14:10:57 -07:00
Doroszlai, Attila
1877312440 HDDS-2238. Container Data Scrubber spams log in empty cluster 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-10-07 13:59:49 -07:00
Surendra Singh Lilhore
382967be51 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-08 00:14:30 +05:30
Doroszlai, Attila
9685a6c0e5 HDDS-2239. Fix TestOzoneFsHAUrls (#1600) 2019-10-07 10:17:25 -07:00
bilaharith
1a77a15fe4
HADOOP-16587. Make ABFS AAD endpoints configurable. 
Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
 2019-10-07 13:07:46 +01:00
dchitlangia
7f332ebf8b
HDDS-2252. Enable gdpr robot test in daily build 
Closes #1602
 2019-10-07 11:35:39 +02:00
pingsutw
14cd969b6e
HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-10-07 16:38:08 +09:00
Shashikant Banerjee
022fe5f5b2 HDDS-2169. Avoid buffer copies while submitting client requests in Ratis. Contributed by Tsz-wo Sze(#1517). 2019-10-07 09:34:36 +05:30
Wei-Chiu Chuang
55c5436f39 Revert "HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár." 
This reverts commit 10bdc592d6.
 2019-10-05 11:16:11 -07:00
Steve Loughran
b8086bf54d
HADOOP-16626. S3A ITestRestrictedReadAccess fails without S3Guard. 
Contributed by Steve Loughran.

Change-Id: Ife730b80057ddd43e919438cb5b2abbda990e636
 2019-10-05 12:52:42 +01:00
Márton Elek
579dc87015
HDDS-2251. Add an option to customize unit.sh and integration.sh parameters 
Closes #1598
 2019-10-05 13:40:02 +02:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Vivek Ratnavel Subramanian
f209722a19 HDDS-2257. Fix checkstyle issues in ChecksumByteBuffer (#1603) 2019-10-04 16:36:32 -07:00
Doroszlai, Attila
a3cf54ccdc
HDDS-2250. Generated configs missing from ozone-filesystem-lib jars 
Closes #1597
 2019-10-04 23:31:22 +02:00
Hanisha Koneru
8de4374427
HDDS-2158. Fixing Json Injection Issue in JsonUtils. (#1486) 2019-10-04 12:52:29 -07:00
avijayanhwx
f3eaa84f9d HDDS-2164 : om.db.checkpoints is getting filling up fast. (#1536) 2019-10-04 12:44:21 -07:00
Norbert Kalmar
10bdc592d6 HADOOP-16579. Upgrade to Apache Curator 4.2.0 excluding ZK (#1531). Contributed by Norbert Kalmár. 2019-10-04 10:57:07 -07:00
Steve Loughran
6574f27fa3
HADOOP-16570. S3A committers encounter scale issues. 
Contributed by Steve Loughran.

This addresses two scale issues which has surfaced in large scale benchmarks
of the S3A Committers.

* Thread pools are not cleaned up.
  This now happens, with tests.

* OOM on job commit for jobs with many thousands of tasks,
  each generating tens of (very large) files.

Instead of loading all pending commits into memory as a single list, the list
of files to load is the sole list which is passed around; .pendingset files are
loaded and processed in isolation -and reloaded if necessary for any
abort/rollback operation.

The parallel commit/abort/revert operations now work at the .pendingset level,
rather than that of individual pending commit files. The existing parallelized
Tasks API is still used to commit those files, but with a null thread pool, so
as to serialize the operations.

Change-Id: I5c8240cd31800eaa83d112358770ca0eb2bca797
 2019-10-04 18:54:22 +01:00
Eric Yang
aa24add8f0 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle
 2019-10-04 13:13:10 -04:00
Elek, Márton
3f166512af HDDS-2237. KeyDeletingService throws NPE if it's started too early (#1584) 2019-10-04 09:48:49 -07:00
Sunil G
4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Doroszlai, Attila
f826420d2b
HDDS-2230. Invalid entries in ozonesecure-mr config. (Addendum) 2019-10-04 17:37:51 +02:00
Tsz-Wo Nicholas Sze
531cc938fe
HDDS-2222. Add a method to update ByteBuffer in PureJavaCrc32/PureJavaCrc32C. (#1595) 2019-10-04 21:16:28 +08:00
Steve Loughran
f44abc3e11
HADOOP-16207 Improved S3A MR tests. 
Contributed by Steve Loughran.

Replaces the committer-specific terasort and MR test jobs with parameterization
of the (now single tests) and use of file:// over hdfs:// as the cluster FS.

The parameterization ensures that only one of the specific committer tests
run at a time -overloads of the test machines are less likely, and so the
suites can be pulled back into the parallel phase.

There's also more detailed validation of the stage outputs of the terasorting;
if one test fails the rest are all skipped. This and the fact that job
output is stored under target/yarn-${timestamp} means failures should
be more debuggable.

Change-Id: Iefa370ba73c6419496e6e69dd6673d00f37ff095
 2019-10-04 14:12:31 +01:00
cxorm
bca014b0e0
HDDS-2216. Rename HADOOP_RUNNER_VERSION to OZONE_RUNNER_VERSION in compose .env files. 
Closes #1570.
 2019-10-04 14:57:33 +02:00
S O'Donnell
6171a41b4c
HDDS-2199. In SCMNodeManager dnsToUuidMap cannot track multiple DNs on the same host 
Closes #1551
 2019-10-04 14:00:06 +02:00
dchitlangia
d061c8469f
HDDS-2140. Add robot test for GDPR feature 
Closes #1542
 2019-10-04 13:51:57 +02:00
Márton Elek
bffcd33085
HDDS-2230. Invalid entries in ozonesecure-mr config 
Doroszlai, Attila <adoroszlai@apache.org>
 2019-10-04 13:24:42 +02:00
Tsz-Wo Nicholas Sze
a9849f65ba
Revert "HDDS-2222 (#1578)" (#1594) 
This reverts commit 4cf0b3660f.
 2019-10-04 19:19:45 +08:00
Tsz-Wo Nicholas Sze
4cf0b3660f
HDDS-2222 (#1578) 
Thanks @jnp  for reviewing this.  Merging now.
 2019-10-04 17:50:21 +08:00
Abhishek Modi
2478cbafe6 YARN-9782. Avoid DNS resolution while running SLS. Contributed by Abhishek Modi. 2019-10-04 14:45:10 +05:30
Takanobu Asanuma
b23bdaf085 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 2019-10-04 16:47:06 +09:00
Doroszlai, Attila
ec8f691201
HDDS-2225. SCM fails to start in most unsecure environments due to leftover secure config 
Closes #1568
 2019-10-04 08:50:13 +02:00
Stephen O'Donnell
c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00