big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,389 Commits 2 Branches 2 Tags 264 MiB
fb92aa4012
Commit Graph

24389 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Jungtaek Lim
f3c629c27e
HADOOP-17388. AbstractS3ATokenIdentifier to issue date in UTC. (#2477) 
Followup to HADOOP-17379.

Contributed by Jungtaek Lim.
 2020-11-20 10:38:42 +00:00
zhang_jf
8fa699b53f
HDFS-15635. ViewFileSystemOverloadScheme support specifying mount table loader imp through conf (#2389). Contributed by Junfan Zhang. 2020-11-19 20:21:16 -08:00
Ahmed Hussein
07050339e0
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 14:37:14 -06:00
Liang-Chi Hsieh
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-11-18 12:03:25 -08:00
Szilard Nemeth
0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth
5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Steve Loughran
ce7827c82a
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. (#2399) 
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs

Contributed by Steve Loughran.

Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
 2020-11-18 13:34:51 +00:00
Steve Loughran
e3c08f285a
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2310) 
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.

In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.

Contributed by Steve Loughran.

Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
 2020-11-18 12:18:11 +00:00
Masatake Iwasaki
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2020-11-18 16:11:09 +09:00
Akira Ajisaka
2045a9df13
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. (#2463) 2020-11-18 14:15:11 +09:00
Jungtaek Lim
a7b923c80c
HADOOP-17379. AbstractS3ATokenIdentifier to set issue date == now. (#2466) 
Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned.

Contributed by Jungtaek Lim.
 2020-11-17 14:43:29 +00:00
Akira Ajisaka
b57f04cd5b
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 2020-11-17 10:56:56 +09:00
Ahmed Hussein
0b2510ee1f
YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 2020-11-16 14:55:40 -06:00
Doroszlai, Attila
dd85a90da6
HADOOP-17376. ITestS3AContractRename failing against stricter tests. (#2462) 
Contributed by Attila Doroszlai.
 2020-11-16 11:24:00 +00:00
Ahmed Hussein
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-13 14:22:35 -06:00
Takanobu Asanuma
f56cd88d7d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 2020-11-13 11:02:26 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
Ahmed Hussein
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals (#2431) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-12 13:13:12 -06:00
Akira Ajisaka
fc961b63d1
Revert "HADOOP-17269. [JDK 11] Upgrade SpotBugs to 4.1.3 to fix false-positive warnings. (#2374)" (#2460) 
This reverts commit 2de7240d4f.
 2020-11-12 15:52:36 +09:00
Doroszlai, Attila
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447) 
Contributed by Attila Doroszlai.
 2020-11-11 21:20:09 +00:00
Ahmed Hussein
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-11 12:39:03 -08:00
Chao Sun
188ebb5a9b
HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade (#2458) 2020-11-11 11:37:23 -06:00
Peter Bacsko
567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
jianghuazhu
375900049c
HDFS-15608.Reset the DistCp#CLEANUP variable definition. (#2351). Contributed by JiangHua Zhu. 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2020-11-10 13:02:29 -08:00
Siyao Meng
fbd2220167
HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448) 2020-11-10 12:13:15 -08:00
Stephen Jung
45434c93e8
HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 2020-11-10 11:38:22 -08:00
Gabriel Medeiros Coelho
61f8c5767e
YARN-10480. replace href tags with ng-href (#2426) 2020-11-10 10:54:28 -08:00
Yizhong Zhang
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 2020-11-10 09:11:31 -08:00
Chao Sun
2522bf2f9b
HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars (#2411) 
Contributed by Chao Sun.
 2020-11-10 16:00:15 +00:00
Akira Ajisaka
4331c88352
HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-11-10 15:19:40 +09:00
Masatake Iwasaki
0946f58cae
HADOOP-17369. Bump up snappy-java to 1.1.8.1. (#2451) 2020-11-10 15:01:24 +09:00
Hui Fei
f1d5652359 HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang. 2020-11-10 13:09:03 +08:00
maobaolong
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 2020-11-10 13:01:10 +08:00
Akira Ajisaka
34e2abd811
HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. (#2433) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-11-10 13:58:27 +09:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
Ahmed Hussein
1ea3f74246
HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-09 14:05:08 -06:00
Steve Loughran
ae7b00a998
HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 09:43:24 +05:30
hchaverr
6eacaffeea HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 2020-11-06 14:25:35 -08:00
Takanobu Asanuma
4b312810ae
Revert "HADOOP-17224. Install Intel ISA-L library in Dockerfile. (#2243)" (#2440) 2020-11-06 11:20:45 +09:00
Eric E Payne
55339c2bdd YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 2020-11-05 21:50:46 +00:00
Eric Badger
af389d9897 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.
 2020-11-05 21:46:14 +00:00
Jim Brennan
638f1fc2b6 Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)" 
This reverts commit d259928035.
 2020-11-05 17:11:39 +00:00
Wei-Chiu Chuang
7e52c0975f Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 7f5caca04c.
 2020-11-04 16:18:03 -08:00
Dongjoon Hyun
2cb9e8f66a
HADOOP-17341. Upgrade commons-codec to 1.15 (#2428) 
Reviewed-by: Chao Sun <sunchao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-11-04 09:12:31 -08:00
He Xiaoqiao
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
 2020-11-04 13:53:46 +08:00
Ayush Saxena
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:28:44 +05:30