big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,091 Commits 2 Branches 2 Tags 264 MiB
e02eb24e0a
Commit Graph

5135 Commits

Author SHA1 Message Date
Steve Loughran
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. 
Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
 2019-06-20 09:56:40 +01:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Shweta Yakkali
dd4a7633ec HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 23:47:57 -07:00
Adam Antal
56c621c68f HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 22:56:00 -07:00
Wei-Chiu Chuang
f1c239c6a4 HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 2019-06-17 21:51:33 -07:00
Wei-Chiu Chuang
10311c30b0 HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong. 2019-06-17 19:55:56 -07:00
Wei-Chiu Chuang
1e92db5a1e HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend. 2019-06-17 18:29:34 -07:00
Gabor Bota
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). 
Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
 2019-06-16 17:05:01 +01:00
cxorm
076618677d HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm. 2019-06-14 20:05:20 -07:00
dineshchitlangia
c7554ffd5c HADOOP-16373. Fix typo in FileSystemShell#test documentation (#968) 2019-06-14 14:36:04 -07:00
Takanobu Asanuma
54f9f75a44 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 2019-06-14 10:17:25 +09:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Shweta Yakkali
d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Erik Krogen
76b94c274f HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 2019-06-07 14:20:44 -07:00
Steve Loughran
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables. 
Contributed by Steve Loughran

Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
 2019-06-07 18:26:10 +01:00
Huan-Ping Su
14552d19e3
HADOOP-16344. Make DurationInfo public unstable. 
Second attempt: imports fixed up.

Contributed by Huan-Ping Su and Kai Xie.

Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
 2019-06-07 12:55:17 +01:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Steve Loughran
1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable." 
This reverts commit 829848ba2e.

Change-Id: Ied91250e191b2ba701a8fc697c78b3756ce76be8
 2019-06-06 17:49:31 +01:00
Huan-Ping Su
829848ba2e
HADOOP-16344. Make DurationInfo public unstable. 
Contributed by Huan-Ping Su

Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
 2019-06-06 12:52:49 +01:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen
d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Steve Loughran
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" 
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
 2019-06-05 13:54:55 +01:00
Steve Loughran
7724d8031b Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors" 
This reverts commit 5906268f0d.
 2019-06-05 13:54:42 +01:00
Steve Loughran
827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch. 
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.

Contributed by Erik Krogen.

Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
 2019-06-04 15:56:47 +01:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Shweta
93d7630062 HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-24 18:42:46 +02:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Eric Yang
ea0b1d8fba HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation. 
Contributed by Prabhu Joseph
 2019-05-23 11:36:32 -04:00
Akira Ajisaka
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang. 2019-05-23 13:59:42 +09:00
Sahil Takiar
5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ben Roling
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. 
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
 2019-05-19 22:29:54 +01:00
Alexis Daboville
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,
 2019-05-17 15:15:22 +01:00
Sahil Takiar
b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)
 2019-05-16 08:57:54 -06:00
David Mollitor
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor
 2019-05-16 16:02:07 +02:00
Bharat Viswanadham
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 2019-05-15 17:41:36 -07:00
Inigo Goiri
389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Akira Ajisaka
f257497b0f HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-09 14:49:46 +09:00
Prabhu Joseph
96dc5cedfe
HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo. 
Author:    Prabhu Joseph
 2019-05-08 10:18:20 +01:00
Peter Bacsko
713e8a27ae HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-07 17:48:27 -07:00
Siyao Meng
93f2283a69 HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-06 15:47:33 -07:00
Vinayakumar B
f1875b205e HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena. 2019-05-03 11:22:14 +05:30
Giovanni Matteo Fumarola
7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Sahil Takiar
4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00
Ben Roling
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure. 2019-04-30 11:53:26 +01:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Anu Engineer
f4ab9370f5 HADOOP-16026:Replace incorrect use of system property user.name. 
Contributed by Dinesh Chitlangia.
 2019-04-22 14:02:13 -07:00
Inigo Goiri
fb1c549139 HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota. 2019-04-22 13:32:08 -07:00