big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,681 Commits 2 Branches 2 Tags 264 MiB
f7a94ec0a4
Commit Graph

5322 Commits

Author SHA1 Message Date
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Wei-Chiu Chuang
8cef9f89f4 HDFS-14652. Addendum: HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-06 15:24:11 -07:00
Eric Yang
22430c10e2 HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager. 
Contributed by Prabhu Joseph
 2019-08-06 17:04:17 -04:00
Jianfei Jiang
71aad60e51
HDFS-14691. Wrong usage hint for hadoop fs command "test". 
Contributed by Jianfei Jiang.

Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
 2019-08-05 13:08:47 +01:00
Wei-Chiu Chuang
61180f4656 HADOOP-15942. Change the logging level form DEBUG to ERROR for RuntimeErrorException in JMXJsonServlet. Contributed by Anuhan Torgonshar. 2019-08-02 14:58:24 -07:00
Wei-Chiu Chuang
797d14e816 Revert "HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm." 
This reverts commit 076618677d.
 2019-08-02 08:25:41 -07:00
Wei-Chiu Chuang
e872ceb810 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 2019-08-01 19:56:51 -07:00
Wei-Chiu Chuang
e20b19543b HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat. 2019-08-01 17:35:31 -07:00
Wei-Chiu Chuang
d086d058d8 HDFS-14652. HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-01 16:13:10 -07:00
Wei-Chiu Chuang
b94eba9f11 HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 2019-08-01 15:50:43 -07:00
He Xiaoqiao
f86de6f76a HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-01 14:54:39 -07:00
Akira Ajisaka
8bda91d20a
HADOOP-16398. Exports Hadoop metrics to Prometheus (#1170) 2019-07-31 10:11:36 -07:00
Wei-Chiu Chuang
0f2dad6679 HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina. 2019-07-30 16:52:42 -07:00
Siyao Meng
c75f16db79 HADOOP-16452. Increase ipc.maximum.data.length default from 64MB to 128MB. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 11:22:45 -07:00
Don Jeba
204a977f55
HADOOP-15910. Fix Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS 
Contributed by Don Jeba.

Change-Id: I2755bfb1263fc659078a1af8f0bdfd739fd1ae40
 2019-07-30 12:39:48 +01:00
Akira Ajisaka
cbfa3f3e98
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 2019-07-29 17:37:26 -07:00
Erik Krogen
62efb63006 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 2019-07-26 11:16:58 -07:00
Gopal V
aebac6d2d2
HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock. 
Contributed by Gopal V.

Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
 2019-07-26 11:32:13 +01:00
Steve Loughran
07530314c2
HADOOP-9844. NPE when trying to create an error message response of SASL RPC 
This closes #55

Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-26 17:53:18 +09:00
sunlisheng
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java. 
This closes #1091

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-24 10:04:39 +09:00
Steve Loughran
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories. 
Contributed by Steve Loughran and Gabor Bota.

This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.

It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".

Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
 2019-07-23 14:52:03 +01:00
S O'Donnell
eb36b09cb7
HADOOP-16443. Improve help text for setfacl --set option. 
Contributed by S O'Donnell.

Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
 2019-07-23 10:24:07 +01:00
Sean Mackrory
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration (#1125) 2019-07-19 09:49:59 -06:00
Josh Rosen
d545f9c290 HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise 
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise 

Contributed by: Josh Rosen
 2019-07-18 23:19:38 +01:00
Gopal V
b4466a3b0a
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 
Contributed by Gopal V and Atilla Magyar.

Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
 2019-07-17 13:50:02 +01:00
Steve Loughran
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3. 
Contributed by Steve Loughran

Contains

- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently

This patch doesn't fix the underlying problem but it

* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones

The outcome is that tests should pass consistently and if problems occur we have better diagnostics.

Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
 2019-07-12 13:02:25 +01:00
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Steve Loughran
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists. 
Contributed by Steve Loughran.

This patch

* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
 2019-07-11 18:15:34 +01:00
Inigo Goiri
aa9f0850e8 HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena. 2019-07-02 13:22:30 -07:00
Christopher Gregorian
129576f628 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 2019-06-24 12:09:17 -07:00
Ajay Kumar
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Ayush Saxena
b52fd05d42 HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Takanobu Asanuma
559cb11551 HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore
f40e10b349 HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Prabhu Joseph
371452e260 HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-21 17:05:13 -07:00
Wei-Chiu Chuang
840d02ca5b HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 19:57:48 -07:00
Steve Loughran
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. 
Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
 2019-06-20 09:56:40 +01:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Shweta Yakkali
dd4a7633ec HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 23:47:57 -07:00
Adam Antal
56c621c68f HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 22:56:00 -07:00
Wei-Chiu Chuang
f1c239c6a4 HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 2019-06-17 21:51:33 -07:00
Wei-Chiu Chuang
10311c30b0 HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong. 2019-06-17 19:55:56 -07:00
Wei-Chiu Chuang
1e92db5a1e HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend. 2019-06-17 18:29:34 -07:00
Gabor Bota
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). 
Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
 2019-06-16 17:05:01 +01:00
cxorm
076618677d HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm. 2019-06-14 20:05:20 -07:00
dineshchitlangia
c7554ffd5c HADOOP-16373. Fix typo in FileSystemShell#test documentation (#968) 2019-06-14 14:36:04 -07:00
Takanobu Asanuma
54f9f75a44 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 2019-06-14 10:17:25 +09:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Shweta Yakkali
d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Erik Krogen
76b94c274f HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 2019-06-07 14:20:44 -07:00
Steve Loughran
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables. 
Contributed by Steve Loughran

Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
 2019-06-07 18:26:10 +01:00
Huan-Ping Su
14552d19e3
HADOOP-16344. Make DurationInfo public unstable. 
Second attempt: imports fixed up.

Contributed by Huan-Ping Su and Kai Xie.

Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
 2019-06-07 12:55:17 +01:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Steve Loughran
1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable." 
This reverts commit 829848ba2e.

Change-Id: Ied91250e191b2ba701a8fc697c78b3756ce76be8
 2019-06-06 17:49:31 +01:00
Huan-Ping Su
829848ba2e
HADOOP-16344. Make DurationInfo public unstable. 
Contributed by Huan-Ping Su

Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
 2019-06-06 12:52:49 +01:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen
d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Steve Loughran
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" 
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
 2019-06-05 13:54:55 +01:00
Steve Loughran
7724d8031b Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors" 
This reverts commit 5906268f0d.
 2019-06-05 13:54:42 +01:00
Steve Loughran
827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch. 
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.

Contributed by Erik Krogen.

Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
 2019-06-04 15:56:47 +01:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Shweta
93d7630062 HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-24 18:42:46 +02:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Eric Yang
ea0b1d8fba HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation. 
Contributed by Prabhu Joseph
 2019-05-23 11:36:32 -04:00
Akira Ajisaka
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang. 2019-05-23 13:59:42 +09:00
Sahil Takiar
5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ben Roling
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. 
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
 2019-05-19 22:29:54 +01:00
Alexis Daboville
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,
 2019-05-17 15:15:22 +01:00
Sahil Takiar
b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)
 2019-05-16 08:57:54 -06:00
David Mollitor
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor
 2019-05-16 16:02:07 +02:00
Bharat Viswanadham
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 2019-05-15 17:41:36 -07:00
Inigo Goiri
389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Akira Ajisaka
f257497b0f HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-09 14:49:46 +09:00
Prabhu Joseph
96dc5cedfe
HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo. 
Author:    Prabhu Joseph
 2019-05-08 10:18:20 +01:00
Peter Bacsko
713e8a27ae HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-07 17:48:27 -07:00
Siyao Meng
93f2283a69 HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-06 15:47:33 -07:00
Vinayakumar B
f1875b205e HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena. 2019-05-03 11:22:14 +05:30
Giovanni Matteo Fumarola
7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Sahil Takiar
4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00
Ben Roling
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure. 2019-04-30 11:53:26 +01:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Anu Engineer
f4ab9370f5 HADOOP-16026:Replace incorrect use of system property user.name. 
Contributed by Dinesh Chitlangia.
 2019-04-22 14:02:13 -07:00
Inigo Goiri
fb1c549139 HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota. 2019-04-22 13:32:08 -07:00
Erik Krogen
1ddb48872f HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 2019-04-22 08:16:57 -07:00
Zsombor Gegesy
008766c119 HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-16 05:28:18 -07:00
Kenneth Yang
b1120d27ab
HADOOP-16249. Make CallerContext LimitedPrivate scope to Public. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-16 11:18:21 +09:00
Gabor Bota
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre. 
Author:    Gabor Bota <gabor.bota@cloudera.com>
 2019-04-12 18:28:38 -07:00
Sahil Takiar
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer. 
Author:    Sahil Takiar <stakiar@cloudera.com>
 2019-04-12 18:12:02 -07:00
Chen Liang
626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Arpit Agarwal
87407553ef
HADOOP-16243. Change Log Level to trace in NetUtils.java. Contributed by chencan. 2019-04-10 13:21:04 -07:00
Giovanni Matteo Fumarola
813cee1a18 HDFS-14420. Fix typo in KeyShell console. Contributed by Hu Xiaodong. 2019-04-10 11:18:40 -07:00
Todd Lipcon
65deb1ac42 HADOOP-16179. hadoop-common pom should not depend on kerb-simplekdc 
The hadoop-common pom currently has a dependency on kerb-simplekdc. In
fact, the only classes used from Kerby are in kerb-core and kerb-util
(which is a transitive dependency frmo kerb-core). Depending on
kerb-simplekdc pulls a bunch of other unnecessary classes into the
hadoop-common classpath.

This changes the hadoop-common pom to depend only on kerb-core.

hadoop-minikdc already had the appropriate dependency on kerb-simplekdc
so it continues to pull in what it needs.

Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-04-10 08:49:46 -07:00
Akira Ajisaka
bb8dda2bf9
HADOOP-12890. Fix typo in AbstractService. Contributed by Gabor Liptak. 2019-04-08 15:26:12 +09:00
Akira Ajisaka
ab645b3caa
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang. 2019-04-08 15:18:45 +09:00
Akira Ajisaka
72f4b9cd68
HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed by Ray Chiang. 2019-04-08 13:20:21 +09:00
Akira Ajisaka
0d47d283a6
HADOOP-10848. Cleanup calling of sun.security.krb5.Config. 2019-04-08 10:02:34 +09:00
David Mollitor
c90736350b
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.
 2019-04-04 16:15:57 +01:00
Inigo Goiri
7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Siyao Meng
e62cbcbc83 HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-03 14:29:52 -07:00
Steve Loughran
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:23:40 +01:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Sahil Takiar
3b0c5016b2
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-04-03 10:56:33 -07:00
Akira Ajisaka
aaaf856f4b
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str 2019-04-03 13:16:59 +09:00
Lokesh Jain
cf268114c9 HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-02 12:24:55 -07:00
Steve Loughran
61d19110d4
HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. (#655) 
Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8
 2019-04-02 09:15:11 +01:00
Akira Ajisaka
ebd0d21538
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 2019-04-02 10:51:47 +09:00
Xiaoyu Yao
f41f938b2e
HADOOP-16199. KMSLoadBlanceClientProvider does not select token correctly. Contributed by Xiaoyu Yao. 
This closes  #642.
 2019-03-28 21:55:31 -07:00
Gabor Bota
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations. 
Author:    Gabor Bota
 2019-03-28 15:59:25 +00:00
David Mollitor
d18d0859eb
HADOOP-16181. HadoopExecutors shutdown Cleanup. 
Author:    David Mollitor <david.mollitor@cloudera.com>
 2019-03-22 10:29:27 +00:00
David Mollitor
246ab77f28
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>
 2019-03-22 10:26:24 +00:00
Eric Yang
5f6e225166 YARN-9363. Replaced debug logging with SLF4J parameterized log message. 
Contributed by Prabhu Joseph
 2019-03-18 13:57:18 -04:00
Eric Yang
5446e3cb8a HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18. 
Contributed by Daniel Templeton
 2019-03-18 13:04:49 -04:00
Erik Krogen
8c95cb9d6b HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 2019-03-18 08:13:43 -07:00
Shweta Yakkali
2db38abffc HDFS-14328. [Clean-up] Remove NULL check before instanceof in TestGSet 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I5b9f0e66664714d7c5bbfa30492a09f770626711
 2019-03-18 15:10:26 +01:00
Eric Yang
2064ca015d YARN-9349. Changed logging to use slf4j api. 
Contributed by Prabhu Joseph
 2019-03-15 19:20:59 -04:00
Ben Roling
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-13 20:37:11 +00:00
Erik Krogen
66357574ae HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 2019-03-13 13:15:56 -07:00
Matt Foley
f74159c8fc HADOOP-16166. TestRawLocalFileSystemContract fails with build Docker container running on Mac. 
Also provided similar fix for Windows.
 2019-03-13 09:33:24 -07:00
Konstantin V Shvachko
2a54feabb2 HDFS-14347. [SBN Read] Restore a comment line mistakenly removed in ProtobufRpcEngine. Contributed by Fengnan Li. 2019-03-11 18:59:15 -07:00
Konstantin V Shvachko
4ad295a4f1 HDFS-14270.[SBN Read] Add trace level logging for stateId in RPC Server. Contributed by Shweta Yakkali. 2019-03-11 13:48:06 -07:00
Steve Loughran
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-09 16:00:34 +00:00
Praveen Krishna
2b94e51a8f
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. 
Author:    Praveen Krishna <praveenkrishna@tutanota.com>
 2019-03-07 11:06:34 +00:00
Vivek Ratnavel Subramanian
a55fc36299 HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values. 2019-03-06 17:43:57 -08:00
Sahil Takiar
618e009ac0 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 15:04:06 -08:00
Wei-Chiu Chuang
6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Eric Yang
3c5b7136e2 HADOOP-16150. Added concat method to ChecksumFS as unsupported operation. 
Contributed by Steve Loughran

(cherry picked from commit 8b517e7ad670aa8ee0b73ce5a572f36ce63eabee)
 2019-03-05 13:32:00 -05:00
Stephen O'Donnell
686c0141ef
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:09:00 +00:00
Prabhu Joseph
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 14:10:08 +09:00
coder-chenzhi
fe7551f21b
HADOOP-16162. Remove unused Job Summary Appender configurations from log4j.properties 
This closes #551

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 13:41:33 +09:00
David Mollitor
9fcd89ab93
HADOOP-16148. Cleanup LineReader Unit Test. 
Contributed by David Mollitor.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-04 23:08:12 +00:00
Ajay Kumar
0d61facd37 HADOOP-15889. Add hadoop.token configuration parameter to load tokens. Contributed by Íñigo Goiri 2019-02-28 10:34:28 -08:00
Márton Elek
84c4966a5a
HADOOP-16067. Incorrect Format Debug Statement KMSACLs. Contributed by Charan Hebri. 2019-02-28 12:15:47 +01:00
Eric Yang
feccd282fe HADOOP-16107. Update ChecksumFileSystem createFile/openFile API to generate checksum. 
Contributed by Steve Loughran
 2019-02-27 15:53:41 -05:00
Tsz Wo Nicholas Sze
9192f71e21 HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 2019-02-26 15:14:21 -08:00
Abhishek Modi
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util. 
Contributed by Abhishek Modi
 2019-02-26 17:10:41 +00:00
Surendra Singh Lilhore
59ba3552d3 HDFS-14299. ViewFs: Correct error message for read only operations. Contributed by hu xiaodong. 2019-02-26 12:40:32 +05:30
Konstantin V Shvachko
a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Inigo Goiri
ba4e7bd192 HADOOP-16125. Support multiple bind users in LdapGroupsMapping. Contributed by Lukas Majercak. 2019-02-25 13:39:13 -08:00
Tsz Wo Nicholas Sze
0edb0c51dc HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections. 2019-02-25 13:15:28 -08:00
Yongjun Zhang
f7a27cdee4 HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li. 2019-02-23 09:35:36 -08:00
George Huang
9daf43c6fa HADOOP-16129. Misc. bug fixes for KMS Benchmark. Contributed by George Huang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-22 17:52:09 -08:00
Daryn Sharp
a87e458432 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 18:17:04 -08:00
Shweta Yakkali
371a6db59a HDFS-14273. Fix checkstyle issues in BlockLocation's method javadoc 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I546aa4a0fe7f83b53735acd9925f366b2f1a00e2
 2019-02-20 15:36:53 -08:00
George Huang
0525d85d57 HADOOP-15967. KMS Benchmark Tool. Contributed by George Huang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-19 15:24:30 -08:00
Akira Ajisaka
dabfeab785
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg. 2019-02-15 14:49:49 +09:00
Erik Krogen
64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Siyao Meng
fd026863b1 HDFS-14241. Provide feedback on successful renameSnapshot and deleteSnapshot. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-13 19:48:02 -08:00
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Vinayakumar B
00c5ffaee2 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:44:32 +05:30
Yiqun Lin
7b11b404a3 HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen. 2019-02-13 11:16:04 +08:00
Xiaoyu Yao
ca4e46a05e HDDS-1075. Fix CertificateUtil#parseRSAPublicKey charsetName. Contributed by Siddharth Wagle. 2019-02-11 12:09:14 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Wangda Tan
308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Steve Loughran
7f46d13dac
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.
 2019-02-01 14:31:47 +00:00
Inigo Goiri
bcc3a79f58 HADOOP-16084. Fix the comment for getClass in Configuration. Contributed by Fengnan Li. 2019-01-31 10:06:05 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Hanisha Koneru
b3bc94ebfd HDFS-14236. Lazy persist copy/ put fails with ViewFs. 2019-01-29 16:45:44 -08:00
Wei-Chiu Chuang
d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00
Pranay Singh
1d523279da HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-25 09:02:57 -08:00
Eric Yang
0dd35e218f HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao
 2019-01-22 18:59:36 -05:00
Akira Ajisaka
a463cf75a0
HADOOP-15787. [JDK11] TestIPC.testRTEDuringConnectionSetup fails. Contributed by Zsolt Venczel. 2019-01-22 10:19:05 +09:00
Sunil G
2e2508b8e3 Make 3.2.0 aware to other branches 2019-01-21 21:24:51 +05:30
Xiaoyu Yao
751bc62df7 Merge branch 'HDDS-4' into trunk 2019-01-18 14:44:24 -08:00
Surendra Singh Lilhore
dacc1a759e HDFS-14175. EC: Native XOR decoder should reset the output buffer before using it. Contributed by Ayush Saxena. 2019-01-18 13:55:12 +05:30
Xiaoyu Yao
570b503e3e Fix merge conflicts 2019-01-15 22:54:50 -07:00
Xiaoyu Yao
ff61931f91 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:16:58 -07:00
Akira Ajisaka
01405df87f
HADOOP-15941. Addendum patch. Contributed by Takanobu Asanuma. 2019-01-16 10:28:08 +09:00
luhuachao
ccc43621ab
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.
 2019-01-15 13:15:18 +00:00
Steve Loughran
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran and Daryn Sharp.
 2019-01-14 17:59:27 +00:00
Inigo Goiri
35fa3bd685 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 2019-01-11 18:07:19 -08:00
Chen Liang
bf08f4abae HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-11 14:01:23 -08:00
Chen Liang
01cb958af4 HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 2019-01-11 12:51:07 -08:00
Giovanni Matteo Fumarola
fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
Akira Ajisaka
d4ca907da6
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 2019-01-11 11:24:05 +09:00
Kihwal Lee
33c009a45d HADOOP-15909. KeyProvider class should implement Closeable. Contributed 
by Kuhu Shukla.
 2019-01-10 13:21:37 -06:00
Jason Lowe
c634589ab2 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit ecdeaa7e6a.
 2019-01-09 17:24:58 -06:00
Takanobu Asanuma
709ddb10e5 HADOOP-15941. [JDK 11] Compilation failure: package com.sun.jndi.ldap is not visible. 2019-01-09 13:55:25 +09:00
shwetayakkali@cloudera.com
4ab5260b7e HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
 2019-01-08 17:04:57 -08:00
Akira Ajisaka
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens." 
This reverts commit d7152332b3.
 2019-01-08 14:51:30 +09:00
Eric Yang
802932ca0b HADOOP-16031. Fixed TestSecureLogins unit test. Contributed by Akira Ajisaka 2019-01-07 13:21:14 -05:00
Steve Loughran
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran.
 2019-01-07 13:18:03 +00:00
Eric Yang
d43af8b3db HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin
 2019-01-04 17:54:15 -05:00
Inigo Goiri
f4e18242bd HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. 2019-01-04 09:56:58 -08:00
Sunil G
51427cbdfb HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 2019-01-04 21:49:06 +05:30
Pranay Singh
ecdeaa7e6a HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-03 09:56:07 -08:00
Akira Ajisaka
57198bc511
HADOOP-15962. The buffer size is small when unpacking tar archives. Contributed by BELUGA BEHR. 2018-12-28 15:38:47 +09:00
Konstantin V Shvachko
bd2a59e508 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2018-12-24 09:39:20 -08:00
Chao Sun
fa8550337d HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
ef3e1929dd HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Chen Liang
cae279610b HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. 2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
cbc2f932eb HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00