Petre Bogdan Stolojan
a218038960
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem ( #3101 )
...
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
for S3
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems
Contributed by: Bogdan Stolojan
2021-07-30 19:42:08 +01:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found ( #3243 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-30 08:29:56 -07:00
Tamas Domok
798a0837c1
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty ( #3206 )
...
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).
Any IO error would have resulted in the same fallback path.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-07-30 12:16:46 +02:00
Gautham B A
13467f4524
HADOOP-17815. Run CI for Centos 7 ( #3231 )
2021-07-29 10:50:57 -07:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. ( #3236 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues ( #3225 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
Szilard Nemeth
74770c8a16
YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao
2021-07-29 17:37:40 +02:00
Szilard Nemeth
54f9fff218
YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao
2021-07-29 13:43:40 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. ( #3194 )
...
YARN-10841. Fix token reset synchronization for UAM response token. Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
jianghuazhu
6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. ( #2836 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-07-28 21:57:28 -07:00
He Xiaoqiao
683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
2021-07-29 11:19:17 +08:00
hchaverr
3c8a48e681
HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. ( #3242 )
2021-07-28 15:37:56 -07:00
Brian Loss
1d03c69963
HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values ( #3221 )
...
Contributed by Brian Loss.
2021-07-28 20:22:58 +01:00
Viraj Jasani
e001f8ee39
HADOOP-17814. Provide fallbacks for identity/cost providers and backoff enable ( #3230 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-29 02:10:07 +09:00
Szilard Nemeth
f2b6c03fc1
YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph
2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9
YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
2021-07-28 16:50:14 +02:00
Mehakmeet Singh
b19dae8db3
HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled ( #3239 )
...
Contributed by Mehakmeet Singh
2021-07-28 15:34:43 +01:00
S O'Donnell
4eae284827
HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell
2021-07-28 14:49:23 +01:00
Szilard Nemeth
8d0297c213
YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
2021-07-28 14:49:10 +02:00
jianghuazhu
fd13970d94
HDFS-16137.Improve the comments related to FairCallQueue#queues. ( #3226 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-07-28 03:18:04 -07:00
bshashikant
dac10fcc20
HDFS-16145. CopyListing fails with FNF exception with snapshot diff. ( #3234 )
2021-07-28 10:29:00 +05:30
9uapaw
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent ( #3212 )
...
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-27 19:35:20 -07:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. ( #3175 )
...
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a
Co-authored-by: Zhihai Xu <zxu@apache.org>
2021-07-27 19:18:44 -07:00
sumangala-patki
10ba4cc892
HADOOP-17765. ABFS: Use Unique File Paths in Tests. ( #3153 )
...
Contributed by Sumangala Patki
2021-07-27 18:49:22 +01:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. ( #3185 )
...
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-27 22:25:21 +05:30
Szilard Nemeth
fa0289b022
YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
2021-07-27 16:47:41 +02:00
Mehakmeet Singh
f813554769
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK ( #2706 )
...
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.
Read the documentation in encryption.md very, very carefully before
use and consider it unstable.
S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":
fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>
You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console.
* Filesystem list/get status operations subtract 16 bytes from the length
of all files >= 16 bytes long to compensate for the padding which CSE
adds.
* The SDK always warns about the specific algorithm chosen being
deprecated. It is critical to use this algorithm for ranged
GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.
Contributed by Mehakmeet Singh
2021-07-27 11:08:51 +01:00
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically ( #3228 ). Contributed by Viraj Jasani.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-27 14:57:12 +08:00
Gautham B A
97c88c97de
HADOOP-17807. Use separate src dir for platform builds ( #3210 )
2021-07-26 13:09:36 -07:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. ( #3229 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web ( #3211 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-26 12:45:01 +08:00
dependabot[bot]
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 ( #2891 )
...
Bumps [http-proxy](https://github.com/http-party/node-http-proxy ) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases )
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-26 02:16:21 +09:00
Akira Ajisaka
2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. ( #3227 )
...
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"
This reverts commit b6d839a60c
.
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
2021-07-26 01:53:39 +09:00
Anoop Sam John
dd8e540670
Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml ( #3223 )
2021-07-25 13:10:27 +05:30
Akira Ajisaka
4c35466359
HADOOP-17317. [JDK 11] Upgrade dnsjava to remove illegal access warnings ( #2442 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-07-24 14:48:25 +09:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. ( #3203 )
...
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
Petre Bogdan Stolojan
63dfd84947
HADOOP-17458. S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException ( #3040 )
...
Some network exceptions can raise SdkClientException with message
`Data read has a different length than the expected`.
These should be recoverable.
Contributed by Bogdan Stolojan
2021-07-23 14:44:29 +01:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor ( #3135 )
...
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135 )
2021-07-23 17:07:51 +05:30
Viraj Jasani
3a52bfc5db
HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException ( #3219 )
2021-07-23 12:38:55 +08:00
Viraj Jasani
98412ce2e1
HADOOP-17813. Checkstyle - Allow line length: 100
...
Signed-off-by: Sean Busbey <busbey@apache.org>
2021-07-22 13:45:49 -05:00
zhuqi-lucas
2da9b95d4d
YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori.
2021-07-22 20:30:43 +08:00
Renukaprasad C
dbd255f4a9
HADOOP-17796. Upgrade jetty version to 9.4.43 ( #3208 )
2021-07-22 01:15:00 -07:00
zhuqi-lucas
0441efe1fc
YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger.
2021-07-21 15:31:44 +08:00
9uapaw
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution ( #3214 )
...
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-21 10:06:46 +08:00
Eric Yin
de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool ( #3141 ). Contributed by Eric Yin.
2021-07-21 00:01:55 +08:00
Viraj Jasani
e1d00addb5
HADOOP-16290. Enable RpcMetrics units to be configurable ( #3198 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-19 23:55:49 -07:00
Masatake Iwasaki
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. ( #3213 )
2021-07-20 11:52:52 +08:00
zhuqi-lucas
0ac443b1f8
YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan.
2021-07-19 10:11:20 +08:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.
2021-07-17 21:57:00 +08:00
Kihwal Lee
47002719f2
HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee.
2021-07-16 14:22:39 -05:00