big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,912 Commits 2 Branches 2 Tags 264 MiB
9be448b336
Commit Graph

22912 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sammi Chen
4222b62f2b HDDS-1879. Support multiple excluded scopes when choosing datanodes in NetworkTopology (#1194) 2019-09-16 12:57:41 -07:00
Márton Elek
c8d61ffef6 HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh scripts 
Signed-off-by: Anu Engineer <aengineer@apache.org>
Co-Authored-By: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
 2019-09-16 12:17:33 -07:00
Xiaoyu Yao
56f042c48f
HDDS-2078. Get/Renew DelegationToken NPE after HDDS-1909 
Closes #1444
 2019-09-16 16:58:10 +02:00
Doroszlai, Attila
363373e0ef
HDDS-2124. Random next links 
Closes #1443
 2019-09-16 15:41:17 +02:00
Doroszlai, Attila
e952ecf807
HDDS-2109. Refactor scm.container.client config 
Closes #1426
 2019-09-16 15:22:11 +02:00
Xiaoyu Yao
b6334381df
HDDS-2096. Ozone ACL document missing AddAcl API 
Closes #1427
 2019-09-16 14:55:06 +02:00
shwetayakkali@cloudera.com
1e13fe6310
HDDS-2044.Remove 'ozone' from the recon module names. 2019-09-16 14:18:01 +02:00
Elek, Márton
85b1c728e4 HDDS-2129. Using dist profile fails with pom.ozone.xml as parent pom (#1449) 2019-09-15 08:22:17 -07:00
Abhishek Modi
da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
Ayush Saxena
e04b8a46c3 HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina. 2019-09-14 10:39:20 +05:30
Nanda kumar
9a931b823e
HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml 
Closes #1439
 2019-09-14 06:32:11 +02:00
Arpit Agarwal
6a9f7caef4
Revert "HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message." 
This reverts commit 95010a41fc.
 2019-09-13 11:40:42 -07:00
Siyao Meng
e22a324f87 HDDS-2007. Make ozone fs shell command work with OM HA service ids (#1360) 2019-09-13 11:22:00 -07:00
Yi Sheng
1843c4688a HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. 2019-09-14 02:11:04 +08:00
avijayanhwx
8024818d33 HDDS-1786 : Datanodes takeSnapshot should delete previously created snapshots (#1163) 2019-09-13 23:20:31 +05:30
Stephen O'Donnell
06ad6540b3 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-13 10:36:45 -07:00
HUAN-PING SU
4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
 2019-09-13 10:36:45 -07:00
sdeka
95010a41fc HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-13 07:08:31 -07:00
Vinayakumar B
39e82acc48 HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429). 
Addendum patch. Moved protobuf-3.7.1 installation within YETUS marker.
 2019-09-13 16:50:16 +05:30
Doroszlai, Attila
4a9a6a21b8
HDDS-2122. Broken logo image on category sub-pages (#1437) 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-09-13 16:11:00 +05:30
Surendra Singh Lilhore
4852a90e4b HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 2019-09-13 00:17:13 +05:30
Gabor Bota
1505d3f5ff
HADOOP-16566. S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch (#1433). Contributed by Gabor Bota. 
Change-Id: Ied43ef1522dfc6a1210d6fc58c38d8208824931b
 2019-09-12 19:04:57 +02:00
Shashikant Banerjee
fe8cdf0ab8 HDDS-2076. Read fails because the block cannot be located in the container (#1410) 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-09-12 21:18:14 +05:30
Surendra Singh Lilhore
2ff2a7f612 HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina. 2019-09-12 19:43:18 +05:30
Surendra Singh Lilhore
d1c303a497 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 2019-09-12 19:11:50 +05:30
Vinayakumar B
f4f9f0fe4f
HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429). Contributed by Vinayakumar B. 2019-09-12 16:47:54 +05:30
Gabor Bota
4e273a31f6
HADOOP-16423. S3Guard fsck: Check metadata consistency between S3 and metadatastore (log) (#1208). Contributed by Gabor Bota. 
Change-Id: I6bbb331b6c0a41c61043e482b95504fda8a50596
 2019-09-12 13:12:46 +02:00
Abhishek Modi
44850f6784 YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph. 2019-09-12 12:50:10 +05:30
Akira Ajisaka
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor. 2019-09-12 12:41:57 +09:00
Abhishek Modi
3b06f0bf9e YARN-9819. Make TestOpportunisticContainerAllocatorAMService more resilient. Contribued by Abhishek Modi 2019-09-12 08:09:57 +05:30
Márton Elek
f537410563
HDDS-2106. Avoid usage of hadoop projects as parent of hdds/ozone 
Closes #1423
 2019-09-12 02:38:41 +02:00
Eric Badger
56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Doroszlai, Attila
64ed6b177d HDDS-2075. Tracing in OzoneManager call is propagated with wrong parent (#1415) 2019-09-11 11:59:01 -07:00
Steve Loughran
9221704f85
HADOOP-16490. Avoid/handle cached 404s during S3A file creation. 
Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,
so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile
in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files
for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even
after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy
independently configurable from other retry policies; it is also exponential, but with
different parameters. This is because every HEAD request will refresh any 404 cached in
the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between
attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s
fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught
so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:
it will fail fast with a different error message. However, because create(path, overwrite=false) no
longer does HEAD path, we can at least be confident that S3A itself is not creating those cached
404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d
 2019-09-11 16:46:25 +01:00
Doroszlai, Attila
5a381f73e9 HDDS-2103. TestContainerReplication fails due to unhealthy container (#1421) 2019-09-11 19:49:10 +05:30
Takanobu Asanuma
c255333e20 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 2019-09-11 16:54:08 +09:00
Akira Ajisaka
dacc44821d
HADOOP-15184. Add GitHub pull request template. (#1419) 2019-09-11 11:10:11 +09:00
Takanobu Asanuma
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414) 2019-09-11 10:32:07 +09:00
Jonathan Hung
f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang
10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. 
Contributed by Prabhu Joseph
 2019-09-10 17:04:39 -04:00
Xieming Li
dc9abd27d9
HADOOP-16554. mvn javadoc:javadoc fails in hadoop-aws. 
Contributed by  Xieming Li.

Change-Id: I78e88b5b1ae4702446d2bdd3e2faa3e10b45aef0
 2019-09-10 15:05:20 +01:00
HUAN-PING SU
38c1a10024 HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-10 19:58:34 +08:00
Lokesh Jain
c3beeb7761 HDDS-2048: State check during container state transition in datanode should be lock protected (#1375) 2019-09-10 14:14:52 +05:30
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Xiaoyu Yao
650c4cead5
HDDS-2053. Fix TestOzoneManagerRatisServer failure. Contributed by Xiaoyu Yao. (#1373) 2019-09-09 15:38:11 -07:00
Mukul Kumar Singh
d69b811ddd HDDS-2102. HddsVolumeChecker should use java optional in place of Guava optional. Contributed by Mukul Kumar Singh. (#1416) 2019-09-09 14:17:28 -07:00
Shashikant Banerjee
469165e6f2 HDDS-1843. Undetectable corruption after restart of a datanode. Contributed by Shashikant Banerjee(#1364). 2019-09-09 22:43:20 +05:30
Sneha Vijayarajan
147f98629c
HADOOP-16438. ADLS Gen1 OpenSSL config control. 
Contributed by Sneha Vijayarajan.

Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d
 2019-09-09 17:09:32 +01:00
Abhishek Modi
60af8793b4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 14:26:00 +05:30
Jonathan Hung
387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00