big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,031 Commits 2 Branches 2 Tags 264 MiB
844b766da5
Commit Graph

23031 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Badger
56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Doroszlai, Attila
64ed6b177d HDDS-2075. Tracing in OzoneManager call is propagated with wrong parent (#1415) 2019-09-11 11:59:01 -07:00
Steve Loughran
9221704f85
HADOOP-16490. Avoid/handle cached 404s during S3A file creation. 
Contributed by Steve Loughran.

This patch avoids issuing any HEAD path request when creating a file with overwrite=true,
so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile
in their own code.

The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files
for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even
after S3A is patched to not do it itself.

Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy
independently configurable from other retry policies; it is also exponential, but with
different parameters. This is because every HEAD request will refresh any 404 cached in
the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between
attempts to (hopefully) ensure any cached entry wil be gone.

The options and values are:

fs.s3a.s3guard.consistency.retry.interval: 2s
fs.s3a.s3guard.consistency.retry.limit: 7

The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught
so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.

Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:
it will fail fast with a different error message. However, because create(path, overwrite=false) no
longer does HEAD path, we can at least be confident that S3A itself is not creating those cached
404 markers.

Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d
 2019-09-11 16:46:25 +01:00
Doroszlai, Attila
5a381f73e9 HDDS-2103. TestContainerReplication fails due to unhealthy container (#1421) 2019-09-11 19:49:10 +05:30
Takanobu Asanuma
c255333e20 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 2019-09-11 16:54:08 +09:00
Akira Ajisaka
dacc44821d
HADOOP-15184. Add GitHub pull request template. (#1419) 2019-09-11 11:10:11 +09:00
Takanobu Asanuma
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414) 2019-09-11 10:32:07 +09:00
Jonathan Hung
f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang
10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. 
Contributed by Prabhu Joseph
 2019-09-10 17:04:39 -04:00
Xieming Li
dc9abd27d9
HADOOP-16554. mvn javadoc:javadoc fails in hadoop-aws. 
Contributed by  Xieming Li.

Change-Id: I78e88b5b1ae4702446d2bdd3e2faa3e10b45aef0
 2019-09-10 15:05:20 +01:00
HUAN-PING SU
38c1a10024 HADOOP-16542. Update commons-beanutils version to 1.9.4. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-10 19:58:34 +08:00
Lokesh Jain
c3beeb7761 HDDS-2048: State check during container state transition in datanode should be lock protected (#1375) 2019-09-10 14:14:52 +05:30
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Xiaoyu Yao
650c4cead5
HDDS-2053. Fix TestOzoneManagerRatisServer failure. Contributed by Xiaoyu Yao. (#1373) 2019-09-09 15:38:11 -07:00
Mukul Kumar Singh
d69b811ddd HDDS-2102. HddsVolumeChecker should use java optional in place of Guava optional. Contributed by Mukul Kumar Singh. (#1416) 2019-09-09 14:17:28 -07:00
Shashikant Banerjee
469165e6f2 HDDS-1843. Undetectable corruption after restart of a datanode. Contributed by Shashikant Banerjee(#1364). 2019-09-09 22:43:20 +05:30
Sneha Vijayarajan
147f98629c
HADOOP-16438. ADLS Gen1 OpenSSL config control. 
Contributed by Sneha Vijayarajan.

Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d
 2019-09-09 17:09:32 +01:00
Abhishek Modi
60af8793b4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 14:26:00 +05:30
Jonathan Hung
387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00
Vivek Ratnavel Subramanian
3b9584d12b HDDS-2087. Remove the hard coded config key in ChunkManager (#1409) 2019-09-08 20:44:07 -07:00
Jonathan Hung
43e389b980 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:07:11 -07:00
Bharat Viswanadham
ca32917e0f
HDDS-2100. Ozone TokenRenewer provider is incorrectly configured. Contributed by Jitendra Nath Pandey. 2019-09-07 12:55:35 -07:00
Abhishek Modi
162af6f554 YARN-9784. org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky. Contributed by Julia Kinga Marton. 2019-09-07 18:08:04 +05:30
Abhishek Modi
e8ca4fac53 YARN-9812. mvn javadoc:javadoc fails in hadoop-sls. Contributed by Abhishek Modi. 2019-09-07 12:44:40 +05:30
Wangda Tan
34b82e6da0 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
 2019-09-06 19:18:13 -07:00
Sammi Chen
c46d43ab13 HDDS-1553. Add metrics in rack aware container placement policy. (#1361) 2019-09-06 17:11:54 -07:00
Tao Yang
73575701ab YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:52:39 +08:00
Tao Yang
5d497abe21 YARN-9817. Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero. Contributed by Prabhu Joseph. 2019-09-07 07:39:02 +08:00
Jungtaek Lim (HeartSaVioR)
bb0b922a71
HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) 
Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
 2019-09-06 21:53:00 +01:00
dchitlangia
b15c116c1e HDDS-2015. Encrypt/decrypt key using symmetric key while writing/reading 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-06 11:35:00 -07:00
Jonathan Hung
b71a7f195c YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 11:00:07 -07:00
Erik Krogen
a23417533e HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang. 2019-09-06 10:28:21 -07:00
Erik Krogen
9637097ef9 HDFS-14817. [Dynamometer] Fix start script options parsing which incorrectly interpret options starting with h as a help argument. Contributed by Soya Miyoshi. 2019-09-06 10:24:51 -07:00
Erik Krogen
c92a3e94d8 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2019-09-06 10:22:28 -07:00
Jonathan Hung
e7d44e48f7 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:17:07 -07:00
Erik Krogen
ae42c8cb61 HDFS-4819. [Dynamometer] Fix parsing of audit logs which contain = in path names. Contributed by Soya Miyoshi. 2019-09-06 10:13:26 -07:00
Surendra Singh Lilhore
d98c54816d HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green. 2019-09-06 18:12:42 +05:30
Surendra Singh Lilhore
fa7f03fc56 HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C. 2019-09-06 17:55:30 +05:30
Lokesh Jain
6e4cdf89ef HDDS-1561: Mark OPEN containers as QUASI_CLOSED as part of Ratis groupRemove (#1401) 2019-09-06 13:15:49 +05:30
Inigo Goiri
494d75eb2b HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang. 2019-09-05 21:15:17 -07:00
Anu Engineer
acbea8d976 HDDS-1708. Add container scrubber metrics. 
Contributed by Hrishikesh Gadre.
 2019-09-05 14:33:06 -07:00
Jonathan Hung
0ccf4b0fe1 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 2019-09-05 14:01:42 -07:00
Billie Rinaldi
2e2e5401f2 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 2019-09-05 12:49:58 -07:00
Inigo Goiri
72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Steve Loughran
511df1e837 HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions 
Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
 2019-09-05 14:25:15 +01:00
Nanda kumar
2b16d5377c
HDDS-1898. GrpcReplicationService#download cannot replicate the container. (#1326) 2019-09-05 18:12:36 +05:30
Tao Yang
172bcd8e01 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. 2019-09-05 16:20:05 +08:00
Hrishikesh Gadre
f347c348d8 Add support for checksum verification in data scrubber 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-04 20:31:49 -07:00
Rohith Sharma K S
3db7184082 YARN-9804. Update ATSv2 document for latest feature supports. 2019-09-05 08:42:17 +05:30
Ayush Saxena
2f70b52a5b HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq. 2019-09-05 08:29:56 +05:30