big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,374 Commits 2 Branches 2 Tags 264 MiB
ebe1d1fbf7
Commit Graph

4377 Commits

Author SHA1 Message Date
Ahmed Hussein
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-13 14:22:35 -06:00
Ahmed Hussein
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals (#2431) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-12 13:13:12 -06:00
Doroszlai, Attila
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447) 
Contributed by Attila Doroszlai.
 2020-11-11 21:20:09 +00:00
Ahmed Hussein
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-11 12:39:03 -08:00
Stephen Jung
45434c93e8
HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 2020-11-10 11:38:22 -08:00
Hui Fei
f1d5652359 HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang. 2020-11-10 13:09:03 +08:00
Ahmed Hussein
1ea3f74246
HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-11-09 14:05:08 -06:00
Steve Loughran
ae7b00a998
HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 09:43:24 +05:30
hchaverr
6eacaffeea HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 2020-11-06 14:25:35 -08:00
Eric Badger
af389d9897 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.
 2020-11-05 21:46:14 +00:00
Jim Brennan
638f1fc2b6 Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)" 
This reverts commit d259928035.
 2020-11-05 17:11:39 +00:00
Wei-Chiu Chuang
7e52c0975f Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 7f5caca04c.
 2020-11-04 16:18:03 -08:00
Akira Ajisaka
7f5caca04c
HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-11-03 11:20:23 -08:00
Peter Bacsko
da93cd962c MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 2020-10-28 10:35:30 +01:00
Ayush Saxena
872440610f
HADOOP-17328. LazyPersist Overwrite fails in direct write mode. (#2413) 2020-10-27 01:37:37 +09:00
Vinayakumar B
d259928035
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:30:02 +05:30
Akira Ajisaka
6a9ceedfb3
HADOOP-17175. [JDK 11] Fix javadoc errors in hadoop-common module. (#2397) 2020-10-23 03:15:45 +09:00
Akira Ajisaka
7b4359657f
HADOOP-17315. Use shaded guava in ClientCache.java (#2398) 
Added checkstyle rules to warn guava imports
 2020-10-20 23:25:24 +09:00
Takanobu Asanuma
30f06e0c74
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 2020-10-20 19:08:59 +09:00
Szilard Nemeth
31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena
4c651103f2
HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2020-10-19 10:54:18 +05:30
hemanthboyina
42e7e0380f HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same 
Contributed by Gabor Bota.
 2020-10-13 16:17:44 +01:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Hui Fei
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 2020-10-09 16:12:22 +08:00
Jinglun
52db86b0bb
HADOOP-17021. Add concat fs command (#1993) 
Contributed by Jinglun
 2020-10-08 10:36:07 +01:00
Mukund Thakur
82522d60fb
HADOOP-17281 Implement FileSystem.listStatusIterator() in S3AFileSystem (#2354) 
Contains HADOOP-17300: FileSystem.DirListingIterator.next() call should 
return NoSuchElementException

Contributed by Mukund Thakur
 2020-10-07 13:59:06 +01:00
Liang-Chi Hsieh
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 
This switches the SnappyCodec to use the java-snappy codec, rather than the native one. 

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh
 2020-10-06 17:07:54 +01:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
maobaolong
d68d2a5c1e
HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:22:55 -07:00
He Xiaoqiao
a490d87eb7 HADOOP-17280. Service-user cost shouldn't be accumulated to totalDecayedCallCost and totalRawCallCost. Contributed by Jinglun. 2020-09-30 12:12:52 +08:00
Xiaoyu Yao
4c5ad57818
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334) 2020-09-29 20:15:27 -07:00
Karen Coppage
75d10f8499
HADOOP-17267. Add debug-level logs in Filesystem.close() (#2321) 
When a filesystem is closed, the FileSystem log will, at debug level,
log the method calling close/closeAll.

At trace level: the full calling stack.

Contributed by Karen Coppage.
 2020-09-29 16:06:50 +01:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00
huangtianhua
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
Hui Fei
474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
Xiaoyu Yao
6b5d9e2334
HADOOP-17259. Allow SSLFactory fallback to input config if ssl-*.xml … (#2301) 2020-09-21 12:41:06 -07:00
crossfire
c8c1cc43d3
HADOOP-17088.Failed to load XInclude files with relative path. (#2097) 
Contributed by Yushi Hayasaka.
 2020-09-21 19:10:51 +01:00
Masatake Iwasaki
364b958085
HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-09-20 00:47:02 +09:00
David Tucker
fc2435cb5c
HADOOP-15136. Correct typos in filesystem.md (#2314) 
Contributed by David Tucker
 2020-09-18 18:28:14 +01:00
Xiaoyu Yao
6adf8462ba
HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances. (#2259) 2020-09-17 10:39:19 -07:00
Uma Maheswara Rao G
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-16 22:43:00 -07:00
Uma Maheswara Rao G
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298). Contributed by Uma Maheswara Rao G. 2020-09-12 17:06:39 -07:00
Steve Loughran
958cab804e
Revert "HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)" 
This reverts commit 9960c01a25.

Change-Id: I820534c3292f2a343693d835f625488c325fb5d6
 2020-09-11 18:07:49 +01:00
1996fanrui
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache (#2241) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-09-10 22:30:52 -07:00
zz
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185). Contributed by Zhenzhao Wang. 
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
 2020-09-10 21:20:32 -07:00
Steve Loughran
9960c01a25
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280) 
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoke.

This avoids directory tombstones being added above files/child directories,
which stop the treewalk and delete phase from working.

Also:

* Callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* New statistic to measure #of objects deleted, alongside request count.
* Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
  to clarify behavior.
* Test enhancements to replicate the failure and verify the fix

Contributed by Steve Loughran
 2020-09-10 17:03:52 +01:00
Takanobu Asanuma
e5fe326270
HADOOP-17165. Implement service-user feature in DecayRPCScheduler. (#2240) 2020-09-09 09:56:58 -07:00