big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,292 Commits 2 Branches 2 Tags 264 MiB
2ee294b1b1
Commit Graph

25292 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bobby Wang
266b1bd1bb
HADOOP-17812. NPE in S3AInputStream read() after failure to reconnect to store (#3222) 
This improves error handling after multiple failures reading data
-when the read fails and attempts to reconnect() also fail.

Contributed by Bobby Wang.
 2021-07-30 20:04:11 +01:00
Petre Bogdan Stolojan
a218038960
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3 
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan
 2021-07-30 19:42:08 +01:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
Tamas Domok
798a0837c1
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty (#3206) 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-07-30 12:16:46 +02:00
Gautham B A
13467f4524
HADOOP-17815. Run CI for Centos 7 (#3231) 2021-07-29 10:50:57 -07:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
Szilard Nemeth
74770c8a16 YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 17:37:40 +02:00
Szilard Nemeth
54f9fff218 YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao 2021-07-29 13:43:40 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
 2021-07-29 11:19:17 +08:00
hchaverr
3c8a48e681 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 2021-07-28 15:37:56 -07:00
Brian Loss
1d03c69963
HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values (#3221) 
Contributed by Brian Loss.
 2021-07-28 20:22:58 +01:00
Viraj Jasani
e001f8ee39
HADOOP-17814. Provide fallbacks for identity/cost providers and backoff enable (#3230) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-29 02:10:07 +09:00
Szilard Nemeth
f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Mehakmeet Singh
b19dae8db3
HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled (#3239) 
Contributed by Mehakmeet Singh
 2021-07-28 15:34:43 +01:00
S O'Donnell
4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
Szilard Nemeth
8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
jianghuazhu
fd13970d94
HDFS-16137.Improve the comments related to FairCallQueue#queues. (#3226) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 03:18:04 -07:00
bshashikant
dac10fcc20
HDFS-16145. CopyListing fails with FNF exception with snapshot diff. (#3234) 2021-07-28 10:29:00 +05:30
9uapaw
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-27 19:35:20 -07:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175) 
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
 2021-07-27 19:18:44 -07:00
sumangala-patki
10ba4cc892
HADOOP-17765. ABFS: Use Unique File Paths in Tests. (#3153) 
Contributed by Sumangala Patki
 2021-07-27 18:49:22 +01:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185) 
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-27 22:25:21 +05:30
Szilard Nemeth
fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
Mehakmeet Singh
f813554769
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706) 
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console. 
  
* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

Contributed by Mehakmeet Singh
 2021-07-27 11:08:51 +01:00
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-27 14:57:12 +08:00
Gautham B A
97c88c97de
HADOOP-17807. Use separate src dir for platform builds (#3210) 2021-07-26 13:09:36 -07:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-26 12:45:01 +08:00
dependabot[bot]
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-26 02:16:21 +09:00
Akira Ajisaka
2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2021-07-26 01:53:39 +09:00
Anoop Sam John
dd8e540670
Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml (#3223) 2021-07-25 13:10:27 +05:30
Akira Ajisaka
4c35466359
HADOOP-17317. [JDK 11] Upgrade dnsjava to remove illegal access warnings (#2442) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-07-24 14:48:25 +09:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203) 
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-24 11:44:21 +08:00
Petre Bogdan Stolojan
63dfd84947
HADOOP-17458. S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException (#3040) 
Some network exceptions can raise SdkClientException with message
`Data read has a different length than the expected`.

These should be recoverable.

Contributed by Bogdan Stolojan
 2021-07-23 14:44:29 +01:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135) 
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
 2021-07-23 17:07:51 +05:30
Viraj Jasani
3a52bfc5db
HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 2021-07-23 12:38:55 +08:00
Viraj Jasani
98412ce2e1 HADOOP-17813. Checkstyle - Allow line length: 100 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-07-22 13:45:49 -05:00
zhuqi-lucas
2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
Renukaprasad C
dbd255f4a9
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3208) 2021-07-22 01:15:00 -07:00
zhuqi-lucas
0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
9uapaw
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution (#3214) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-21 10:06:46 +08:00
Eric Yin
de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
Viraj Jasani
e1d00addb5
HADOOP-16290. Enable RpcMetrics units to be configurable (#3198) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-19 23:55:49 -07:00
Masatake Iwasaki
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 2021-07-20 11:52:52 +08:00
zhuqi-lucas
0ac443b1f8 YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan. 2021-07-19 10:11:20 +08:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00