big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,735 Commits 2 Branches 2 Tags 264 MiB
37ced452d9
Commit Graph

24735 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
1e59bf7394
HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure (#2473). 
Contributed by Steve Loughran

The addition of deprecated S3A configuration options in HADOOP-17318
triggered a reload of default (xml resource) configurations, which breaks
tests which fail if there's a per-bucket setting inconsistent with test
setup.

Creating an S3AFS instance before creating the Configuration() instance
for test runs gets that reload out the way before test setup takes
place.

Along with the fix, extra changes in the failing test suite to fail
fast when marker policy isn't as expected, and to log FS state better.

Rather than create and discard an instance, add a new static method
to S3AFS and invoke it in test setup. This forces the load

Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
 2020-11-26 17:28:01 +00:00
Steve Loughran
1eeb9d9d67
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. (#2399) 
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs

Contributed by Steve Loughran.

Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
 2020-11-26 17:22:56 +00:00
Sneha Vijayarajan
c48c774d6c
HADOOP-17397. ABFS: SAS Test updates for version and permission update (#2492) 
Contributed by Sneha Vijayarajan.

Change-Id: I89c1061b1efb1e3bef019dd22f221d03bf015929
 2020-11-26 10:21:37 +00:00
Sneha Vijayarajan
39fa2c93c4
HADOOP-17396. ABFS: testRenameFileOverExistingFile fails (#2491) 
Contributed by Sneha  Vijayarajan.

Change-Id: I57a866b95ff18229caee8a6028874074a29cb5bd
 2020-11-26 10:13:55 +00:00
Akira Ajisaka
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287)
 2020-11-26 11:34:39 +09:00
Steve Loughran
1ef34d0819
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.

Change-Id: I57161b026f28349e339dc8b9d74f6567a62ce196
 2020-11-25 14:55:29 +00:00
bilaharith
b8454a4b10
HADOOP-17311. ABFS: Logs should redact SAS signature (#2422) 
Contributed by bilaharith.

Change-Id: Iff0ed4303ac5ce41b62bfda8150ee983dafa40be
 2020-11-25 14:33:29 +00:00
Peter Bacsko
8ed565382f MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan. 2020-11-25 11:36:58 +01:00
Mukund Thakur
9dd74141a6
HADOOP-17323. S3A getFileStatus("/") to skip IO (#2479) 
Contributed by Mukund Thakur.

Change-Id: I1709ad72b829999b6dd324f0755b51bc38918d30
 2020-11-24 11:34:19 +00:00
Akira Ajisaka
bb4c6db9b3
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7df)
 2020-11-24 15:22:51 +09:00
Steve Loughran
38cc47d308
HADOOP-17332. S3A MarkerTool -min and -max are inverted. (#2425) 
This patch
* fixes the inversion
* adds a precondition check
* if the commands are supplied inverted, swaps them with a warning.
  This is to stop breaking any tests written to cope with the existing
  behavior.

Contributed by Steve Loughran

Change-Id: I15c40863f0db0675c7d60db477cb3bf1693cae49
 2020-11-23 21:49:33 +00:00
Eric Payne
8459f1d955 HADOOP-17346. Fair call queue is defeated by abusive service principals. Contributed by Ahmed Hussein (ahussein). 2020-11-23 20:37:33 +00:00
Steve Loughran
7ca539bc1b
HADOOP-17325. WASB Test Failures 
Contributed by Ayush Saxena and Steve Loughran

Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
 2020-11-23 17:25:58 +00:00
dengzh
abc87aef18
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469) 
Contributed by Zhihua Deng.

Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
 2020-11-23 16:33:41 +00:00
Steve Loughran
e4bc64cce0 HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468) 
Contributed by Steve Loughran.
 2020-11-23 14:09:14 +00:00
Peter Bacsko
ced08fd87f MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:14:49 +01:00
Jungtaek Lim
401cadbac5
HADOOP-17388. AbstractS3ATokenIdentifier to issue date in UTC. (#2477) 
Followup to HADOOP-17379.

Contributed by Jungtaek Lim.

Change-Id: I7b2fce36028d297c1e095499691a08caba92d9fd
 2020-11-20 10:56:57 +00:00
Jim Brennan
e24a6b550e HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 21:26:47 +00:00
Steve Loughran
4687c25389 HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2310) 
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.

In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.

Contributed by Steve Loughran.

Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
 2020-11-18 12:30:43 +00:00
Steve Loughran
4bb9d593da
HADOOP-17261. s3a rename() needs s3:deleteObjectVersion permission (#2303) 
Contributed by Steve Loughran.

Change-Id: I8e89a402a24bd9fb958e0fa93d1a28191093851d
 2020-11-18 12:20:12 +00:00
Akira Ajisaka
71c106b412
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. (#2463) 
(cherry picked from commit 2045a9df13)
 2020-11-18 14:16:45 +09:00
Jungtaek Lim
22039a14ff
HADOOP-17379. AbstractS3ATokenIdentifier to set issue date == now. (#2466) 
Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned.

Contributed by Jungtaek Lim.

Change-Id: Ic80ac7895612a1aa669459c73a78a9c17ecf0c0d
 2020-11-17 14:56:58 +00:00
Akira Ajisaka
be108159b1
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 
(cherry picked from commit b57f04cd5b)
 2020-11-17 11:01:23 +09:00
Ahmed Hussein
df4edb99f7 HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 1ea3f74246)
 2020-11-16 21:48:37 +00:00
Ahmed Hussein
bc634c077b YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1f)
 2020-11-16 21:09:36 +00:00
Doroszlai, Attila
bf2ff35a04
HADOOP-17376. ITestS3AContractRename failing against stricter tests. (#2462) 
Contributed by Attila Doroszlai.

Change-Id: Ie15624ec07b1c5e34ca7fde0a72a54431d79e746
 2020-11-16 11:26:06 +00:00
Ahmed Hussein
75ca0c0f23 HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit ebe1d1fbf7)
 2020-11-13 21:14:47 +00:00
Takanobu Asanuma
d78bf82722 HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d)
 2020-11-13 11:05:20 +09:00
Ahmed Hussein
23fe3bdab3 HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0f)
 2020-11-12 10:35:28 -08:00
Doroszlai, Attila
47131cdf7c
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447) 
Contributed by Attila Doroszlai.

Change-Id: I21c29256b52449b7fea335704b3afa02e39c6a39
 2020-11-11 21:21:11 +00:00
Chao Sun
8e4b1cd192 HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade (#2458) 2020-11-11 09:49:20 -08:00
Chao Sun
64344fb302 HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars (#2411) 
Contributed by Chao Sun.
 2020-11-11 09:48:57 -08:00
Stephen Jung
0712505b59 HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 
(cherry picked from commit 45434c93e8)
 2020-11-10 11:41:21 -08:00
Gabriel Medeiros Coelho
9a3f2bb545 YARN-10480. replace href tags with ng-href (#2426) 
(cherry picked from commit 61f8c5767e)
 2020-11-10 11:00:00 -08:00
Yizhong Zhang
12bfd9194a HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794)
 2020-11-10 09:14:12 -08:00
Peter Bacsko
0361837c9e YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko. 2020-11-10 13:16:52 +01:00
Akira Ajisaka
aa3807ed1d
HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. (#2433) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 34e2abd811)
 2020-11-10 14:00:08 +09:00
Steve Loughran
7cb5325dda HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 17:05:23 +05:30
hchaverr
043cca01b1 HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 
(cherry picked from commit 6eacaffeea)
 2020-11-06 14:31:31 -08:00
Eric E Payne
1e22929885 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bdd)
 2020-11-05 22:08:51 +00:00
Eric Badger
c6fee0a2c8 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.

(cherry picked from commit af389d9897)
 2020-11-05 21:56:46 +00:00
Jim Brennan
41d58d190d Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)" 
This reverts commit e21b81276e.
 2020-11-05 17:31:39 +00:00
Wei-Chiu Chuang
cfa0986d00 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit dd1634ec3b.
 2020-11-04 16:18:23 -08:00
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 14:52:05 -08:00
Akira Ajisaka
dd1634ec3b HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04c)
 2020-11-03 11:22:48 -08:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518)
 2020-10-29 19:52:43 +00:00
Sunil G
91a3d298b9 HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d52)
 2020-10-29 07:20:46 +05:30
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743)
 2020-10-28 10:33:41 -07:00