Commit Graph

5638 Commits

Author SHA1 Message Date
Ayush Saxena
ad40715690 HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun. 2020-12-07 15:00:19 +05:30
Hui Fei
e2c1268ebd HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 09:20:09 +08:00
Ahmed Hussein
8c234fc0d4
HADOOP-17389. KMS should log full UGI principal. (#2476)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
2020-12-04 09:33:30 +09:00
Attila Magyar
db73e994ed HADOOP-16881. KerberosAuthentication does not disconnect HttpURLConnection leading to CLOSE_WAIT cnxns. Contributed by Attila Magyar.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-03 12:05:20 -08:00
Ahmed Hussein
f94e927bfb
HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-03 10:55:51 -06:00
Andrea Scarpino
9170eb566b
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512)
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-12-04 00:14:04 +09:00
Gautham B A
44910b53f3
HDFS-15699 Remove lz4 references in vcxproj (#2498) 2020-11-29 22:53:54 -08:00
Akira Ajisaka
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:32:10 +09:00
Akira Ajisaka
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488)" because JIRA ID is missing.
This reverts commit ce5b3d7975.
2020-11-26 11:30:53 +09:00
Akira Ajisaka
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs (#2488)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:27:30 +09:00
Steve Loughran
ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.
2020-11-25 14:31:02 +00:00
zhang_jf
8fa699b53f
HDFS-15635. ViewFileSystemOverloadScheme support specifying mount table loader imp through conf (#2389). Contributed by Junfan Zhang. 2020-11-19 20:21:16 -08:00
Ahmed Hussein
07050339e0
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 14:37:14 -06:00
Liang-Chi Hsieh
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec (#2350)
Contributed by Liang-Chi Hsieh.
2020-11-18 12:03:25 -08:00
Steve Loughran
ce7827c82a
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. (#2399)
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs

Contributed by Steve Loughran.

Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
2020-11-18 13:34:51 +00:00
Steve Loughran
e3c08f285a
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2310)
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.

In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.

Contributed by Steve Loughran.

Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
2020-11-18 12:18:11 +00:00
Ahmed Hussein
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-13 14:22:35 -06:00
Ahmed Hussein
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals (#2431)
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-12 13:13:12 -06:00
Doroszlai, Attila
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447)
Contributed by Attila Doroszlai.
2020-11-11 21:20:09 +00:00
Ahmed Hussein
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations (#2436)
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-11 12:39:03 -08:00
Stephen Jung
45434c93e8
HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 2020-11-10 11:38:22 -08:00
Hui Fei
f1d5652359 HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang. 2020-11-10 13:09:03 +08:00
Ahmed Hussein
1ea3f74246
HADOOP-17360. Log the remote address for authentication success (#2441)
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-09 14:05:08 -06:00
Steve Loughran
ae7b00a998
HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 09:43:24 +05:30
hchaverr
6eacaffeea HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 2020-11-06 14:25:35 -08:00
Eric Badger
af389d9897 HADOOP-17342. Creating a token identifier should not do kerberos name
resolution. Contributed by Jim Brennan.
2020-11-05 21:46:14 +00:00
Jim Brennan
638f1fc2b6 Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)"
This reverts commit d259928035.
2020-11-05 17:11:39 +00:00
Wei-Chiu Chuang
7e52c0975f Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)"
This reverts commit 7f5caca04c.
2020-11-04 16:18:03 -08:00
Akira Ajisaka
7f5caca04c
HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-11-03 11:20:23 -08:00
Sunil G
f17e067d52 HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 2020-10-29 07:19:49 +05:30
Peter Bacsko
da93cd962c MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 2020-10-28 10:35:30 +01:00
Ayush Saxena
872440610f
HADOOP-17328. LazyPersist Overwrite fails in direct write mode. (#2413) 2020-10-27 01:37:37 +09:00
Vinayakumar B
d259928035
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:30:02 +05:30
Akira Ajisaka
6a9ceedfb3
HADOOP-17175. [JDK 11] Fix javadoc errors in hadoop-common module. (#2397) 2020-10-23 03:15:45 +09:00
Akira Ajisaka
7b4359657f
HADOOP-17315. Use shaded guava in ClientCache.java (#2398)
Added checkstyle rules to warn guava imports
2020-10-20 23:25:24 +09:00
Takanobu Asanuma
30f06e0c74
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 2020-10-20 19:08:59 +09:00
Szilard Nemeth
31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena
4c651103f2
HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2020-10-19 10:54:18 +05:30
hemanthboyina
42e7e0380f HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
He Xiaoqiao
ddc0ee27fa HADOOP-17304. KMS ACL: Allow DeleteKey Operation to Invalidate Cache. Contributed by Xiaoyu.
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-10-14 16:00:37 +08:00
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
Contributed by Gabor Bota.
2020-10-13 16:17:44 +01:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Hui Fei
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 2020-10-09 16:12:22 +08:00
Jinglun
52db86b0bb
HADOOP-17021. Add concat fs command (#1993)
Contributed by Jinglun
2020-10-08 10:36:07 +01:00
Mukund Thakur
82522d60fb
HADOOP-17281 Implement FileSystem.listStatusIterator() in S3AFileSystem (#2354)
Contains HADOOP-17300: FileSystem.DirListingIterator.next() call should 
return NoSuchElementException

Contributed by Mukund Thakur
2020-10-07 13:59:06 +01:00
Liang-Chi Hsieh
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec (#2297)
This switches the SnappyCodec to use the java-snappy codec, rather than the native one. 

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh
2020-10-06 17:07:54 +01:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
maobaolong
d68d2a5c1e
HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:22:55 -07:00
He Xiaoqiao
a490d87eb7 HADOOP-17280. Service-user cost shouldn't be accumulated to totalDecayedCallCost and totalRawCallCost. Contributed by Jinglun. 2020-09-30 12:12:52 +08:00
Xiaoyu Yao
4c5ad57818
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334) 2020-09-29 20:15:27 -07:00
Karen Coppage
75d10f8499
HADOOP-17267. Add debug-level logs in Filesystem.close() (#2321)
When a filesystem is closed, the FileSystem log will, at debug level,
log the method calling close/closeAll.

At trace level: the full calling stack.

Contributed by Karen Coppage.
2020-09-29 16:06:50 +01:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G.
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-25 21:21:01 -07:00
huangtianhua
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189)
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
2020-09-24 14:27:04 +05:30
Hui Fei
474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322)
Contributed by Hui Fei.
2020-09-22 17:10:33 +01:00
Xiaoyu Yao
6b5d9e2334
HADOOP-17259. Allow SSLFactory fallback to input config if ssl-*.xml … (#2301) 2020-09-21 12:41:06 -07:00
crossfire
c8c1cc43d3
HADOOP-17088.Failed to load XInclude files with relative path. (#2097)
Contributed by Yushi Hayasaka.
2020-09-21 19:10:51 +01:00
Masatake Iwasaki
364b958085
HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-09-20 00:47:02 +09:00
David Tucker
fc2435cb5c
HADOOP-15136. Correct typos in filesystem.md (#2314)
Contributed by David Tucker
2020-09-18 18:28:14 +01:00
Xiaoyu Yao
6adf8462ba
HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances. (#2259) 2020-09-17 10:39:19 -07:00
Uma Maheswara Rao G
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G.
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-16 22:43:00 -07:00
Uma Maheswara Rao G
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298). Contributed by Uma Maheswara Rao G. 2020-09-12 17:06:39 -07:00
Steve Loughran
958cab804e
Revert "HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)"
This reverts commit 9960c01a25.

Change-Id: I820534c3292f2a343693d835f625488c325fb5d6
2020-09-11 18:07:49 +01:00
1996fanrui
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache (#2241)
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-09-10 22:30:52 -07:00
zz
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185). Contributed by Zhenzhao Wang.
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
2020-09-10 21:20:32 -07:00
Steve Loughran
9960c01a25
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoke.

This avoids directory tombstones being added above files/child directories,
which stop the treewalk and delete phase from working.

Also:

* Callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* New statistic to measure #of objects deleted, alongside request count.
* Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
  to clarify behavior.
* Test enhancements to replicate the failure and verify the fix

Contributed by Steve Loughran
2020-09-10 17:03:52 +01:00
Takanobu Asanuma
e5fe326270
HADOOP-17165. Implement service-user feature in DecayRPCScheduler. (#2240) 2020-09-09 09:56:58 -07:00
Steve Loughran
2029556dbb
HADOOP-17181. Handle transient stream read failures in FileSystem contract tests (#2286)
Contributed by Steve Loughran.

* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient

The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.
2020-09-09 11:58:26 +01:00
Bharat Viswanadham
8a3952436c
HADOOP-17245. Add OzoneFileSystem classes to core-default.xml. (#2276) 2020-09-04 18:38:48 -07:00
Uma Maheswara Rao G
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well (#2234). Contributed by Uma Maheswara Rao G. 2020-09-03 11:06:20 -07:00
He Xiaoqiao
7e671f3b65 HADOOP-17235. Erasure Coding: Remove dead code from common side. Contributed by Fei Hui. 2020-09-01 11:35:55 +08:00
Steve Loughran
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing (#2252)
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
2020-08-28 15:39:09 +01:00
Ayush Saxena
bfd99f31fd HADOOP-17232. Erasure Coding: Typo in document. Contributed by Fei Hui. 2020-08-28 17:38:23 +05:30
sguggilam
d8aaa8c338
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249)
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2020-08-26 23:45:21 -07:00
Mingliang Liu
5e52955112
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)"
This reverts commit a932796d0c.
2020-08-26 10:41:10 -07:00
sguggilam
a932796d0c
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2020-08-24 23:39:57 -07:00
Joey
960fb0aa4f
HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896)
Contributed by Jiayi Liu
2020-08-24 14:01:48 +01:00
S O'Donnell
17cd8a1b16 HADOOP-17209. Erasure Coding: Native library memory leak. Contriubted by Sean Chow 2020-08-24 12:00:35 +01:00
Uma Maheswara Rao G
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 2020-08-19 09:30:41 -07:00
Steve Loughran
5092ea62ec HADOOP-13230. S3A to optionally retain directory markers.
This adds an option to disable "empty directory" marker deletion,
so avoid throttling and other scale problems.

This feature is *not* backwards compatible.
Consult the documentation and use with care.

Contributed by Steve Loughran.

Change-Id: I69a61e7584dc36e485d5e39ff25b1e3e559a1958
2020-08-15 12:51:08 +01:00
Akira Ajisaka
141c62584b
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 2020-08-12 16:08:36 +09:00
Siyao Meng
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory (#2176) 2020-08-11 08:52:16 -07:00
Gautham B A
909f1e82d3
HADOOP-17196. Fix C/C++ standard warnings (#2208)
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-08-11 16:35:08 +09:00
Uma Maheswara Rao G
8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205)
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test
2020-08-11 00:01:58 -07:00
sguggilam
2986058e7f
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178)
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2020-08-04 10:30:06 -07:00
Akira Ajisaka
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors (#2098) 2020-08-03 10:46:51 +09:00
belugabehr
e60096c377
HADOOP-17141. Add Capability To Get Text Length (#2157)
Contributed by David Mollitor
2020-07-24 10:37:28 +01:00
Akira Ajisaka
d5b4766158
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 2020-07-23 00:39:49 +09:00
Uma Maheswara Rao G
ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 2020-07-21 23:29:10 -07:00
Masatake Iwasaki
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Chen Liang
3833c616e0 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 2020-07-20 12:49:58 -07:00
Ayush Saxena
f2033de234 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 2020-07-20 22:13:48 +05:30
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Mukund Thakur
b21cb91c7f
HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142)
Contributed by Mukund Thakur
2020-07-16 18:09:59 +01:00
Ahmed Hussein
1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 11:39:06 -05:00
Erik Krogen
317fe4584a HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-14 11:22:25 -07:00
Hanisha Koneru
e62d8f8412 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 2020-07-13 12:55:34 -07:00
Steve Loughran
b9fa5e0182
HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext.
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
2020-07-13 13:30:02 +01:00
lfengnan
84b74b335c
HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110) 2020-07-12 00:10:12 -07:00
Uma Maheswara Rao G
3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 2020-07-11 23:50:04 -07:00
Siyao Meng
0e694b20b9
HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 2020-07-09 12:38:52 -07:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Uma Maheswara Rao G
dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 2020-07-06 18:50:03 -07:00
Madhusoodan Pataki
2f500e4635
HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089)
Contributed by Madhusoodan P
2020-07-06 16:25:42 +01:00
Uma Maheswara Rao G
1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-07-04 00:12:10 -07:00
Abhishek Das
3b8d0f803f
HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 12:58:35 +05:30
Virajith Jalaparti
bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100)
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle
2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G
d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-06-26 01:29:38 -07:00
lfengnan
84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch (#2047)
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
2020-06-23 13:12:29 -07:00
He Xiaoqiao
fa14e4bc00 HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 2020-06-23 16:59:51 +08:00
Uma Maheswara Rao G
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 2020-06-23 01:42:25 -07:00
Uma Maheswara Rao G
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 2020-06-20 00:32:02 -07:00
belugabehr
100ec8e870
HADOOP-17009: Embrace Immutability of Java Collections 2020-06-19 10:23:43 -07:00
Mehakmeet Singh
2bfb22840a
HADOOP-17020. Improve RawFileSystem Performance (#2063)
Contributed by : Mehakmeet Singh

Co-authored-by: Rajesh Balamohan
Co-authored-by: Mehakmeet Singh
2020-06-17 16:15:26 +01:00
Ayush Saxena
c8ed33cd2a HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor. 2020-06-17 13:55:40 +05:30
Uma Maheswara Rao G
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 2020-06-12 14:32:19 -07:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G
93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 2020-06-10 15:00:02 -07:00
Mingliang Liu
0c25131ca4
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:28:56 -07:00
Mingliang Liu
9f242c215e
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:11:30 -07:00
Ayush Saxena
a8610c15c4
HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 2020-06-07 13:29:10 -07:00
Uma Maheswara Rao G
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 2020-06-06 08:11:57 -07:00
Abhishek Das
e7dd02768b
HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 2020-06-05 14:56:51 -07:00
Mike
97c98ce531
HADOOP-14566. Add seek support for SFTP FileSystem. (#1999)
Contributed by Mikhail Pryakhin
2020-06-03 11:37:40 +01:00
Xiaoyu Yao
6288e15118 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 2020-06-02 11:53:32 -07:00
Uma Maheswara Rao G
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00
Dhiraj
9fe4c37c25
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036)
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-06-01 10:49:17 -07:00
S O'Donnell
19f26a020e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 2020-05-29 14:36:56 +01:00
S O'Donnell
d9e8046a1a HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 10:32:37 +01:00
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Uma Maheswara Rao G
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 2020-05-21 21:34:58 -07:00
Akira Ajisaka
52b21de1d8
HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027)
* Removed javax.activation-api from dependency
2020-05-22 11:19:11 +09:00
Kei Kori
1a3c6bb33b
HDFS-15353. Use sudo instead of su to allow nologin user for secure DataNode (#2018) 2020-05-21 10:07:23 +09:00
Abhishek Das
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 2020-05-18 22:27:12 -07:00
Akira Ajisaka
27601fc79e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 2020-05-18 15:36:20 +09:00
Uma Maheswara Rao G
ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 2020-05-14 17:29:35 -07:00
Mike
017d24e970
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists.
Contributed by Mikhail Pryakhin.
2020-05-14 18:28:00 +01:00
Joseph Smith
d60496e6c6
BytesWritable causes OOME when array size reaches Integer.MAX_VALUE. (#393) 2020-05-13 00:20:35 +05:30
Ayush Saxena
936bf09c37 HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 2020-05-12 21:54:54 +05:30
Elixir Kook
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007)
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
S O'Donnell
29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Akira Ajisaka
328eae9a14
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003) 2020-05-11 14:44:18 +09:00
Akira Ajisaka
c784ba370e
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 2020-05-08 17:18:23 +09:00
Mingliang Liu
e9e1ead089
HADOOP-17027. Add tests for reading fair call queue capacity weight configs. Contributed by Fengnan Li 2020-05-07 16:50:23 -07:00
Uma Maheswara Rao G
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 2020-05-04 17:55:40 -07:00
Mingliang Liu
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2020-04-30 14:15:28 -07:00
Ayush Saxena
6bdab3723e HADOOP-16957. NodeBase.normalize doesn't removing all trailing slashes. Contributed by Ayush Saxena. 2020-04-30 19:55:20 +05:30
S O'Donnell
9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
lfengnan
4202750040
HADOOP-17010. Add queue capacity support for FairCallQueue (#1977) 2020-04-28 16:14:55 -07:00
Mike
18d7dfbf35
HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952)
Contributed by Mikhail Pryakhin.
2020-04-27 14:43:51 +01:00
Dhiraj
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964)
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:04:32 -07:00
Ayush Saxena
ef9a6e775c HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-04-25 13:07:32 +05:30
Mingliang Liu
af85971a58
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:36:33 -07:00
Steve Loughran
42711081e3
HADOOP-16986. S3A to not need wildfly on the classpath. (#1948)
HADOOP-16986. S3A to not need wildfly JAR on its classpath.

Contributed by Steve Loughran

This is a successor to HADOOP-16346, which enabled the S3A connector
to load the native openssl SSL libraries for better HTTPS performance.

That patch required wildfly.jar to be on the classpath. This
update:

* Makes wildfly.jar optional except in the special case that 
"fs.s3a.ssl.channel.mode" is set to "openssl"

* Retains the declaration of wildfly.jar as a compile-time
dependency in the hadoop-aws POM. This means that unless
explicitly excluded, applications importing that published
maven artifact will, transitively, add the specified
wildfly JAR into their classpath for compilation/testing/
distribution.

This is done for packaging and to offer that optional
speedup. It is not mandatory: applications importing
the hadoop-aws POM can exclude it if they choose.
2020-04-20 14:32:13 +01:00
Ayush Saxena
79e03fb622 HADOOP-16971. TestFileContextResolveAfs#testFileContextResolveAfs creates dangling link and fails for subsequent runs. Contributed by Ctest. 2020-04-20 01:59:18 +05:30
Masatake Iwasaki
ac40daece1
HADOOP-16972. Ignore AuthenticationFilterInitializer for KMSWebServer. (#1961) 2020-04-18 06:38:25 +09:00
belugabehr
eca05917d6
HADOOP-16951: Tidy Up Text and ByteWritables Classes.
1. Remove superfluous code
2. Remove superfluous comments
3. Checkstyle fixes
4. Remove methods that simply call super.method()
5. Use Java 8 facilities to streamline code where applicable
6. Simplify and unify some of the constructs between the two classes
7. Expanding of the arrays be 1.5x instead of 2x per expansion.
2020-04-17 08:16:12 -07:00
Ayush Saxena
3edbe8708a HADOOP-16958. NPE when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL. Contributed by Ctest. 2020-04-14 02:13:42 +05:30
Ayush Saxena
2ea5adf2de HADOOP-16967. TestSequenceFile#testRecursiveSeqFileCreate fails in subsequent run. Contributed by Ctest. 2020-04-12 11:31:07 +05:30
Ayush Saxena
5746533cde HADOOP-16962. Making getBoolean log warning message for unrecognized value. Contributed by Ctest. 2020-04-09 23:30:28 +05:30
Ayush Saxena
1189af4746 HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena. 2020-04-08 15:55:14 +05:30
Takanobu Asanuma
b89d875f7b HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 2020-04-06 11:03:10 +09:00
Masatake Iwasaki
8f8be6b92a HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh Radhakrishnan. 2020-04-04 07:20:51 +09:00
He Xiaoqiao
d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-04-01 16:36:46 -07:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Chao Sun
e3fbdcbc14 HADOOP-16912. Emit per priority RPC queue time and processing time from DecayRpcScheduler. Contributed by Fengnan Li. 2020-03-25 10:21:20 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Wei-Chiu Chuang
f197f05cff
HADOOP-16661. Support TLS 1.3 (#1880) 2020-03-16 10:56:30 -07:00
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
Steve Loughran
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
2020-03-09 14:44:28 +00:00
Sebastian Nagel
18050bc583
HADOOP-16909 Typo in distcp counters.
Contributed by Sebastian Nagel.
2020-03-09 14:37:08 +00:00
Steve Loughran
d0a7c790c6
HADOOP-16885. Fix hadoop-commons TestCopy failure
Followup to HADOOP-16885: Encryption zone file copy failure leaks a temp file

Moving the delete() call broke a mocking test, which slipped through the review process.

Contributed by Steve Loughran.

Change-Id: Ia13faf0f4fffb1c99ddd616d823e4f4d0b7b0cbb
2020-03-03 17:25:22 +00:00
cpugputpu
5678b19b01
HADOOP-16897. Sort fields in ReflectionUtils.java.
Contributed by cpugputpu.
2020-03-02 17:53:38 +00:00
Xiaoyu Yao
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file
Contributed by Xiaoyu Yao.

Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails

Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.

This is a convergence of a fix for S3 meeting an issue in HDFS.

S3: a HEAD against a file can cache a 404, 
 -you must not do any existence checks, including deleteOnExit(),
  until the file is written. 

Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct. 

HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.

This patch:

* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
2020-03-02 13:22:00 +00:00
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Takanobu Asanuma
5cbc4c5461 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 2020-02-25 11:08:13 +09:00
Sahil Takiar
42dfd270a1
HADOOP-16859: ABFS: Add unbuffer support to ABFS connector.
Contributed by Sahil Takiar
2020-02-24 16:28:00 +00:00
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Wei-Chiu Chuang
cb3f3cca01 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 2020-02-18 21:53:08 -08:00
Ayush Saxena
ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Arpit Agarwal
0cfff16ac0
HADOOP-16833. InstrumentedLock should log lock queue time. Contributed by Stephen O'Donnell.
Change-Id: Idddff05051b6f642b88e51694b40c5bb1bef0026
2020-02-18 09:50:11 -08:00
Steve Loughran
a562942b05
HADOOP-16759. FileSystem Javadocs to list what breaks on API changes
Followup to the main openFile().withStatus() patch.
It turns out that this broke the hive builds, which
was not well appreciated.

This patch lists places to review in the hadoop codebase,
and external projects where changes are likely to cause problems.

Contributed by Steve Loughran

Change-Id: Ifac815c65b74d083cd277764b780ac2b5b0f6b36
2020-02-17 22:14:39 +00:00
Ayush Saxena
84f7638840 HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri. 2020-02-18 00:43:33 +05:30
Akira Ajisaka
954930e9d9
HADOOP-16850. Support getting thread info from thread group for JvmMetrics to improve the performance. Contributed by Tao Yang. 2020-02-14 15:20:28 +09:00
Steve Loughran
56dee66770
HADOOP-16823. Large DeleteObject requests are their own Thundering Herd.
Contributed by Steve Loughran.

During S3A rename() and delete() calls, the list of objects delete is
built up into batches of a thousand and then POSTed in a single large
DeleteObjects request.

But as the IO capacity allowed on an S3 partition may only be 3500 writes
per second *and* each entry in that POST counts as a single write, then
one of those posts alone can trigger throttling on an already loaded
S3 directory tree. Which can trigger backoff and retry, with the same
thousand entry post, and so recreate the exact same problem.

Fixes

* Page size for delete object requests is set in
  fs.s3a.bulk.delete.page.size; the default is 250.
* The property fs.s3a.experimental.aws.s3.throttling (default=true)
  can be set to false to disable throttle retry logic in the AWS
  client SDK -it is all handled in the S3A client. This
  gives more visibility in to when operations are being throttled
* Bulk delete throttling events are logged to the log
  org.apache.hadoop.fs.s3a.throttled log at INFO; if this appears
  often then choose a smaller page size.
* The metric "store_io_throttled" adds the entire count of delete
  requests when a single DeleteObjects request is throttled.
* A new quantile, "store_io_throttle_rate" can track throttling
  load over time.
* DynamoDB metastore throttle resilience issues have also been
  identified and fixed. Note: the fs.s3a.experimental.aws.s3.throttling
  flag does not apply to DDB IO precisely because there may still be
  lurking issues there and it safest to rely on the DynamoDB client
  SDK.

Change-Id: I00f85cdd94fc008864d060533f6bd4870263fd84
2020-02-13 19:09:49 +00:00
Stephen O'Donnell
d7c136b9ed HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-11 08:00:15 -08:00
Jan Hentschel
cc8ae59104
HADOOP-16851. Removed unused import in Configuration
Contributed by Jan Hentschel.
2020-02-11 11:51:45 +00:00
testfixer
d36cd37e60
HADOOP-16847. Test can fail if HashSet iterates in a different order.
Contributed by Testfixer
2020-02-11 11:22:07 +00:00
Ayush Saxena
23787e4bdd HDFS-15136. LOG flooding in secure mode when Cookies are not set in request header. Contributed by Renukaprasad C 2020-02-08 01:17:59 +05:30
Akira Ajisaka
3ebf505965
HADOOP-16834. Replace com.sun.istack.Nullable with javax.annotation.Nullable in DNS.java. Contributed by Xieming Li. 2020-02-07 19:30:06 +09:00
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Chen Liang
ce7b8b5634 HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:12:35 -08:00
belugabehr
1afd54fbbb HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor. 2020-01-25 20:42:21 +05:30
Steve Loughran
839e6076c1 HADOOP-16827. TestHarFileSystem.testInheritedMethodsImplemented broken. (#1817)
This is a regression caused by HADOOP-16759.

The test TestHarFileSystem uses introspection to verify that HarFileSystem
Does not implement methods to which there is a suitable implementation in
the base FileSystem class. Because of the way it checks this, refactoring
(protected) FileSystem methods in an IDE do not automatically change
the probes in TestHarFileSystem.

The changes in HADOOP-16759 did exactly that, and somehow managed
to get through the build/test process without this being noticed.

This patch fixes that failure.

Caused by and fixed by Steve Loughran.

Change-Id: If60d9c97058242871c02ad1addd424478f84f446

Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-01-24 10:50:16 -08:00
Mustafa Iman
839054754b
HADOOP-16792: Make S3 client request timeout configurable.
Contributed by Mustafa Iman.

This adds a new configuration option fs.s3a.connection.request.timeout
to declare the time out on HTTP requests to the AWS service;
0 means no timeout.
Measured in seconds; the usual time suffixes are all supported

Important: this is the maximum duration of any AWS service call,
including upload and copy operations. If non-zero, it must be larger
than the time to upload multi-megabyte blocks to S3 from the client,
and to rename many-GB files. Use with care.

Change-Id: I407745341068b702bf8f401fb96450a9f987c51c
2020-01-24 13:37:07 +00:00
Mingliang Liu
6c1fa24ac0 HADOOP-16732. S3Guard to support encrypted DynamoDB table (#1752). Contributed by Mingliang Liu. 2020-01-23 14:21:42 +01:00
Zhankun Tang
d40d7cc4f9 HADOOP-16670. Stripping Submarine code from Hadoop codebase. Contributed by Zhankun Tang.
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-01-21 20:07:31 -08:00
Steve Loughran
5e2ce370a3 HADOOP-16759. Filesystem openFile() builder to take a FileStatus param (#1761). Contributed by Steve Loughran
* Enhanced builder + FS spec
* s3a FS to use this to skip HEAD on open
* and to use version/etag when opening the file

works with S3AFileStatus FS and S3ALocatedFileStatus
2020-01-21 14:31:51 -08:00
Sahil Takiar
f206b736f0
HADOOP-16346. Stabilize S3A OpenSSL support.
Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`.
The new option is documented and marked as experimental.

For details on how to use this, consult the peformance document
in the s3a documentation.

This patch is the successor to HADOOP-16050 "S3A SSL connections
should use OpenSSL" -which was reverted because of
incompatibilities between the wildfly OpenSSL client and the AWS
HTTPS servers (HADOOP-16347). With the Wildfly release moved up
to 1.0.7.Final (HADOOP-16405) everything should now work.

Related issues:

* HADOOP-15669. ABFS: Improve HTTPS Performance
* HADOOP-16050: S3A SSL connections should use OpenSSL
* HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8
* HADOOP-16405: Upgrade Wildfly Openssl version to 1.0.7.Final

Contributed by Sahil Takiar

Change-Id: I80a4bc5051519f186b7383b2c1cea140be42444e
2020-01-21 16:37:51 +00:00