big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,455 Commits 2 Branches 2 Tags 264 MiB
4c033bafa0
Commit Graph

24455 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Siyao Meng
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472) 2020-11-25 11:01:04 -08:00
Steve Loughran
ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.
 2020-11-25 14:31:02 +00:00
bilaharith
3193d8c793
HADOOP-17311. ABFS: Logs should redact SAS signature (#2422) 
Contributed by bilaharith.
 2020-11-25 14:22:10 +00:00
Ankit Kumar
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-25 13:00:09 +09:00
Ahmed Hussein
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions (#2461) 2020-11-24 11:09:30 -08:00
Szilard Nemeth
f813f14b5b MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan 2020-11-24 17:02:35 +01:00
Mukund Thakur
5fee95076b
HADOOP-17323. S3A getFileStatus("/") to skip IO (#2479) 
Contributed by Mukund Thakur.
 2020-11-24 11:06:56 +00:00
Akira Ajisaka
c4ba0ab7df
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 2020-11-24 15:22:07 +09:00
Steve Loughran
9b4faf2b51
HADOOP-17332. S3A MarkerTool -min and -max are inverted. (#2425) 
This patch
* fixes the inversion
* adds a precondition check
* if the commands are supplied inverted, swaps them with a warning.
  This is to stop breaking any tests written to cope with the existing
  behavior.

Contributed by Steve Loughran
 2020-11-23 20:49:42 +00:00
Steve Loughran
07b7d07388
HADOOP-17325. WASB Test Failures 
Contributed by Ayush Saxena and Steve Loughran

Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
 2020-11-23 17:22:13 +00:00
dengzh
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469) 
Contributed by Zhihua Deng.
 2020-11-23 15:40:22 +00:00
Steve Loughran
fb79be932c
HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468) 
Contributed by Steve Loughran.
 2020-11-23 14:08:12 +00:00
He Xiaoqiao
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh
d73029463c
HDFS-15690. Add lz4-java as test dependency (#2481) 2020-11-21 17:49:56 -08:00
dengzh
740399a602
HADOOP-17390. Skip license check on lz4 code files (#2478) 
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
 2020-11-20 14:12:20 +00:00
Ahmed Hussein
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2020-11-20 18:03:45 +05:30
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Jungtaek Lim
f3c629c27e
HADOOP-17388. AbstractS3ATokenIdentifier to issue date in UTC. (#2477) 
Followup to HADOOP-17379.

Contributed by Jungtaek Lim.
 2020-11-20 10:38:42 +00:00
zhang_jf
8fa699b53f
HDFS-15635. ViewFileSystemOverloadScheme support specifying mount table loader imp through conf (#2389). Contributed by Junfan Zhang. 2020-11-19 20:21:16 -08:00
Ahmed Hussein
07050339e0
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 14:37:14 -06:00
Liang-Chi Hsieh
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-11-18 12:03:25 -08:00
Szilard Nemeth
0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth
5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Steve Loughran
ce7827c82a
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. (#2399) 
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs

Contributed by Steve Loughran.

Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
 2020-11-18 13:34:51 +00:00
Steve Loughran
e3c08f285a
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2310) 
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.

In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.

Contributed by Steve Loughran.

Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
 2020-11-18 12:18:11 +00:00
Masatake Iwasaki
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2020-11-18 16:11:09 +09:00
Akira Ajisaka
2045a9df13
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. (#2463) 2020-11-18 14:15:11 +09:00
Jungtaek Lim
a7b923c80c
HADOOP-17379. AbstractS3ATokenIdentifier to set issue date == now. (#2466) 
Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned.

Contributed by Jungtaek Lim.
 2020-11-17 14:43:29 +00:00
Akira Ajisaka
b57f04cd5b
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 2020-11-17 10:56:56 +09:00
Ahmed Hussein
0b2510ee1f
YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 2020-11-16 14:55:40 -06:00
Doroszlai, Attila
dd85a90da6
HADOOP-17376. ITestS3AContractRename failing against stricter tests. (#2462) 
Contributed by Attila Doroszlai.
 2020-11-16 11:24:00 +00:00
Ahmed Hussein
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-13 14:22:35 -06:00
Takanobu Asanuma
f56cd88d7d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 2020-11-13 11:02:26 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
Ahmed Hussein
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals (#2431) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-12 13:13:12 -06:00
Akira Ajisaka
fc961b63d1
Revert "HADOOP-17269. [JDK 11] Upgrade SpotBugs to 4.1.3 to fix false-positive warnings. (#2374)" (#2460) 
This reverts commit 2de7240d4f.
 2020-11-12 15:52:36 +09:00
Doroszlai, Attila
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447) 
Contributed by Attila Doroszlai.
 2020-11-11 21:20:09 +00:00
Ahmed Hussein
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-11 12:39:03 -08:00
Chao Sun
188ebb5a9b
HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade (#2458) 2020-11-11 11:37:23 -06:00
Peter Bacsko
567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
jianghuazhu
375900049c
HDFS-15608.Reset the DistCp#CLEANUP variable definition. (#2351). Contributed by JiangHua Zhu. 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2020-11-10 13:02:29 -08:00
Siyao Meng
fbd2220167
HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448) 2020-11-10 12:13:15 -08:00
Stephen Jung
45434c93e8
HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 2020-11-10 11:38:22 -08:00
Gabriel Medeiros Coelho
61f8c5767e
YARN-10480. replace href tags with ng-href (#2426) 2020-11-10 10:54:28 -08:00
Yizhong Zhang
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 2020-11-10 09:11:31 -08:00
Chao Sun
2522bf2f9b
HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars (#2411) 
Contributed by Chao Sun.
 2020-11-10 16:00:15 +00:00
Akira Ajisaka
4331c88352
HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-11-10 15:19:40 +09:00