big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,538 Commits 2 Branches 2 Tags 264 MiB
d29007fb35
Commit Graph

5015 Commits

Author SHA1 Message Date
Steve Loughran
e25a5c2eab HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by Steve Loughran. 2019-08-09 15:52:37 +02:00
Szilard Nemeth
df30d8ea09 YARN-9727: Allowed Origin pattern is discouraged if regex contains *. Contributed by Zoltan Siegl 2019-08-09 09:34:23 +02:00
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Wei-Chiu Chuang
8cef9f89f4 HDFS-14652. Addendum: HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-06 15:24:11 -07:00
Eric Yang
22430c10e2 HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager. 
Contributed by Prabhu Joseph
 2019-08-06 17:04:17 -04:00
Jianfei Jiang
71aad60e51
HDFS-14691. Wrong usage hint for hadoop fs command "test". 
Contributed by Jianfei Jiang.

Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
 2019-08-05 13:08:47 +01:00
Wei-Chiu Chuang
61180f4656 HADOOP-15942. Change the logging level form DEBUG to ERROR for RuntimeErrorException in JMXJsonServlet. Contributed by Anuhan Torgonshar. 2019-08-02 14:58:24 -07:00
Wei-Chiu Chuang
797d14e816 Revert "HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm." 
This reverts commit 076618677d.
 2019-08-02 08:25:41 -07:00
Wei-Chiu Chuang
e872ceb810 HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko. 2019-08-01 19:56:51 -07:00
Wei-Chiu Chuang
d086d058d8 HDFS-14652. HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang. 2019-08-01 16:13:10 -07:00
Wei-Chiu Chuang
b94eba9f11 HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 2019-08-01 15:50:43 -07:00
He Xiaoqiao
f86de6f76a HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-01 14:54:39 -07:00
Akira Ajisaka
8bda91d20a
HADOOP-16398. Exports Hadoop metrics to Prometheus (#1170) 2019-07-31 10:11:36 -07:00
Wei-Chiu Chuang
0f2dad6679 HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina. 2019-07-30 16:52:42 -07:00
Siyao Meng
c75f16db79 HADOOP-16452. Increase ipc.maximum.data.length default from 64MB to 128MB. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 11:22:45 -07:00
Akira Ajisaka
cbfa3f3e98
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 2019-07-29 17:37:26 -07:00
Erik Krogen
62efb63006 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 2019-07-26 11:16:58 -07:00
Gopal V
aebac6d2d2
HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock. 
Contributed by Gopal V.

Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
 2019-07-26 11:32:13 +01:00
Steve Loughran
07530314c2
HADOOP-9844. NPE when trying to create an error message response of SASL RPC 
This closes #55

Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-26 17:53:18 +09:00
sunlisheng
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java. 
This closes #1091

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-24 10:04:39 +09:00
Steve Loughran
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories. 
Contributed by Steve Loughran and Gabor Bota.

This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.

It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".

Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
 2019-07-23 14:52:03 +01:00
S O'Donnell
eb36b09cb7
HADOOP-16443. Improve help text for setfacl --set option. 
Contributed by S O'Donnell.

Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
 2019-07-23 10:24:07 +01:00
Sean Mackrory
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration (#1125) 2019-07-19 09:49:59 -06:00
Josh Rosen
d545f9c290 HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise 
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise 

Contributed by: Josh Rosen
 2019-07-18 23:19:38 +01:00
Gopal V
b4466a3b0a
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 
Contributed by Gopal V and Atilla Magyar.

Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
 2019-07-17 13:50:02 +01:00
Steve Loughran
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3. 
Contributed by Steve Loughran

Contains

- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently

This patch doesn't fix the underlying problem but it

* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones

The outcome is that tests should pass consistently and if problems occur we have better diagnostics.

Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
 2019-07-12 13:02:25 +01:00
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Steve Loughran
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists. 
Contributed by Steve Loughran.

This patch

* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
 2019-07-11 18:15:34 +01:00
Inigo Goiri
aa9f0850e8 HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena. 2019-07-02 13:22:30 -07:00
Christopher Gregorian
129576f628 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 2019-06-24 12:09:17 -07:00
Ajay Kumar
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Ayush Saxena
b52fd05d42 HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Takanobu Asanuma
559cb11551 HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore
f40e10b349 HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Prabhu Joseph
371452e260 HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-21 17:05:13 -07:00
Wei-Chiu Chuang
840d02ca5b HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 19:57:48 -07:00
Steve Loughran
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. 
Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
 2019-06-20 09:56:40 +01:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Shweta Yakkali
dd4a7633ec HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 23:47:57 -07:00
Adam Antal
56c621c68f HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 22:56:00 -07:00
Wei-Chiu Chuang
10311c30b0 HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong. 2019-06-17 19:55:56 -07:00
Wei-Chiu Chuang
1e92db5a1e HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend. 2019-06-17 18:29:34 -07:00
Gabor Bota
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). 
Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
 2019-06-16 17:05:01 +01:00
cxorm
076618677d HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm. 2019-06-14 20:05:20 -07:00
dineshchitlangia
c7554ffd5c HADOOP-16373. Fix typo in FileSystemShell#test documentation (#968) 2019-06-14 14:36:04 -07:00
Takanobu Asanuma
54f9f75a44 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 2019-06-14 10:17:25 +09:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Shweta Yakkali
d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00