big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,437 Commits 2 Branches 2 Tags 264 MiB
45b1c86fe5
Commit Graph

27437 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
13c5825c00
YARN-11573. Add config option to make container allocation prefer nodes without reserved containers (#6098) 2023-09-22 20:00:50 +02:00
K0K0V0K
0780710f25
YARN-11567 - Aggregate container launch debug artifacts on error (#6053) 2023-09-22 15:09:17 +02:00
huhaiyang
cc66683b1a
HDFS-17184. Improve BlockReceiver to throws DiskOutOfSpaceException when initialize. (#6044). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-21 21:45:30 +08:00
Viraj Jasani
27cb551821
HADOOP-18829. S3A prefetch LRU cache eviction metrics (#5893) 
Contributed by: Viraj Jasani
 2023-09-21 14:31:44 +05:30
slfan1989
42b8e6faa7
YARN-11570. Add YARN_GLOBALPOLICYGENERATOR_HEAPSIZE to yarn-env for GPG. (#6086) 2023-09-20 17:11:59 -07:00
Jian Zhang
d273c13ab5
HDFS-17198. RBF: fix bug of getRepresentativeQuorum when records have same dateModified (#6096) 2023-09-20 10:04:29 -07:00
Syed Shameerur Rahman
5512c9f924
HADOOP-18797. Support Concurrent Writes With S3A Magic Committer (#6006) 
Jobs which commit their work to S3 thr
magic committer now use a unique magic
containing the job ID:
 __magic_job-${jobid}

This allows for multiple jobs to write
to the same destination simultaneously.

Contributed by Syed Shameerur Rahman
 2023-09-20 11:26:42 +01:00
Pranav Saxena
f24b73e5f3
HADOOP-18873. ABFS: AbfsOutputStream doesnt close DataBlocks object. (#6010) 
AbfsOutputStream to close the dataBlock object created for the upload.

Contributed By: Pranav Saxena
 2023-09-20 14:24:36 +05:30
ConfX
39f36d9071
HDFS-17105. mistakenly purge editLogs even after it is empty in NNStorageRetentionManager. (#6036). Contributed by ConfX. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-20 11:46:57 +08:00
slfan1989
512e39e89d
Revert "MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng." (#6103) 
This reverts commit ab2bc90e09.
 2023-09-20 08:04:05 +08:00
PJ Fanning
c16484ffb2
HADOOP-18890. Remove use of okhttp in runtime code (#6057) 
Contributed by PJ Fanning
 2023-09-19 12:38:36 +01:00
Hexiaoqiao
23c22b2823
HADOOP-18906. Increase default batch size of ZKDTSM token seqnum to reduce overflow speed of zonde dataVersion. (#6097) 2023-09-18 10:50:53 -07:00
章锡平
60f3a2b101
HDFS-17138 RBF: We changed the hadoop.security.auth_to_local configur… (#5921) 2023-09-18 09:40:22 -07:00
hfutatzhanghb
18f9989ff2
HDFS-17192. Add bock info when constructing remote block reader meets IOException. (#6081). Contributed by farmmamba. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-18 20:56:45 +08:00
zhangshuyan
9e489b9ab5
HDFS-17190. EC: Fix bug of OIV processing XAttr. (#6067). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-18 10:35:13 +08:00
Steve Loughran
120620c1b7
HADOOP-18888. S3A. createS3AsyncClient() always enables multipart uploads (#6056) 
* The multipart flag fs.s3a.multipart.uploads.enabled is passed to the async client created
* s3A connector bypasses the transfer manager entirely if disabled or for small files.

Contributed by Steve Loughran
 2023-09-15 15:45:09 +01:00
zhtttylz
510a7dcae0
HDFS-17180. HttpFS Add Support getTrashRoots API (#6028) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-15 21:01:24 +08:00
zhengchenyu
ab2bc90e09
MAPREDUCE-7453. Container logs are missing when yarn.app.container.log.filesize is set to default value 0. (#6042) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-15 20:50:11 +08:00
Vikas Kumar
e283375cdf
HADOOP-18851: Performance improvement for DelegationTokenSecretManager. (#6001). Contributed by Vikas Kumar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-15 12:32:47 +08:00
ConfX
23360b3f6b
HADOOP-18824. ZKDelegationTokenSecretManager causes ArithmeticException due to improper numRetries value checking (#6052) 2023-09-14 15:53:31 -07:00
PJ Fanning
dea446419f
HADOOP-18895. Upgrade to commons-compress 1.24.0 (#6062) 
Contributed by PJ Fanning
 2023-09-14 17:49:12 +01:00
slfan1989
8538af4638
YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator. (#5934) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Co-authored-by: slfan1989 <slfan1989@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-14 21:28:49 +08:00
PJ Fanning
56b928b86f
YARN-11498. Add exclusion for jettison everywhere jersey-json is loaded (#5786) 
All uses  of jersey-json in the yarn and other hadoop modules now
exclude the obsolete org.codehaus.jettison/jettison and so avoid
all security issues which can come from the library.

Contributed by PJ Fanning
 2023-09-13 18:10:24 +01:00
slfan1989
9c8cdbe28e
YARN-11433. Router's main() should support generic options. (#6012) 2023-09-12 14:48:40 -07:00
slfan1989
475932c524
YARN-11562. [Federation] GPG Support Query Policies In Web. (#6023) 2023-09-12 14:47:37 -07:00
dannytbecker
4652d22b91
HDFS-17178: BootstrapStandby needs to handle RollingUpgrade (#6018) 2023-09-12 14:46:08 -07:00
Jian Zhang
e4dd7fe1b7
HDFS-17139. RBF: For the doc of the class RouterAdminProtocolTranslatorPB, it describes the function of the class ClientNamenodeProtocolTranslatorPB (#5917) 2023-09-12 14:44:55 -07:00
Steve Loughran
81d90fd65b
HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995) 
This patch migrates the S3A connector to use the V2 AWS SDK.

This is a significant change at the source code level.
Any applications using the internal extension/override points in
the filesystem connector are likely to break.

This includes but is not limited to:
- Code invoking methods on the S3AFileSystem class
  which used classes from the V1 SDK.
- The ability to define the factory for the `AmazonS3` client, and
  to retrieve it from the S3AFileSystem. There is a new factory
  API and a special interface S3AInternals to access a limited
  set of internal classes and operations.
- Delegation token and auditing extensions.
- Classes trying to integrate with the AWS SDK.

All standard V1 credential providers listed in the option 
fs.s3a.aws.credentials.provider will be automatically remapped to their
V2 equivalent.

Other V1 Credential Providers are supported, but only if the V1 SDK is
added back to the classpath.  

The SDK Signing plugin has changed; all v1 signers are incompatible.
There is no support for the S3 "v2" signing algorithm.

Finally, the aws-sdk-bundle JAR has been replaced by the shaded V2
equivalent, "bundle.jar", which is now exported by the hadoop-aws module.

Consult the document aws_sdk_upgrade for the full details.

Contributed by Ahmar Suhail + some bits by Steve Loughran
 2023-09-11 14:30:25 +01:00
slfan1989
e4550e1aef
YARN-6476. Advanced Federation UI based on YARN UI v2. (#5920) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-11 13:28:31 +08:00
huhaiyang
3bd6a751ed
HDFS-17177. ErasureCodingWork should ignore the deleted block while reconstructing blocks (#6024) 2023-09-11 11:57:52 +08:00
zhengchenyu
c5e9510b54
YARN-8980. Mapreduce application container start fail after AM restart. (#5975) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-09 09:50:53 +08:00
zhengchenyu
bf605c8acc
YARN-11564. Fix wrong config in yarn-default.xml (#6030) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-08 15:23:26 +08:00
slfan1989
3ab983fbae
YARN-11434. [Router] UGI conf doesn't read user overridden configurations on Router startup. (#6011) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-08 07:52:54 +08:00
Szilard Nemeth
9342ecf6cc HADOOP-18870. CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709. Contributed by Ferenc Erdelyi 2023-09-06 21:32:36 -04:00
dannytbecker
07c8b69f7c
HDFS-17167. Add config to startup NameNode as Observer (#6013) 2023-09-06 09:17:12 -07:00
slfan1989
e1dde3bc23
YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#5944) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-06 22:08:59 +08:00
Liangjun He
c2c6972f25
HDFS-17140. Revisit the BPOfferService.reportBadBlocks() method. (#5924). Contributed by Liangjun He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 20:19:04 +08:00
ZanderXu
7c941e00b4
HDFS-16933. A race in SerialNumberMap will cause wrong owner, group and XATTR. (#5430). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 10:27:52 +08:00
Jian Zhang
c40a6bd46a
HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover (#5990) 2023-09-05 17:48:27 -07:00
huhaiyang
2831c7ce26
HADOOP-18880. Add some rpc related metrics to Metrics.md (#6015) Contributed by Yanghai Hu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-05 17:34:05 +08:00
slfan1989
a6c2526c6c
YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf. (#5967) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-03 20:08:40 +08:00
slfan1989
d5334fa761
YARN-6537. Running RM tests against the Router. (#5957) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-03 15:38:21 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS (#5992) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-02 15:24:52 +08:00
slfan1989
b6d06c8928
YARN-11554. Fix TestRMFailover#testWebAppProxyInStandAloneMode Failed. (#5971) 2023-08-31 10:46:33 -07:00
Yang Jiandan
2f739450be
YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store (#5978) Contributed by Jiandan Yang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-01 00:14:46 +08:00
Anmol Asrani
01cc6d0bc8
HADOOP-18865. ABFS: Add "100-continue" in userAgent if enabled (#5987) 
Contributed by Anmol Asrani
 2023-08-31 15:10:04 +01:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
WangYuanben
39eaf8d14e
YARN-11556. Let Federation.md more standardized. (#5983) 2023-08-29 12:16:10 +08:00
yuyanlei
b58885624b
HDFS-17093. Fix block report lease issue to avoid missing some storages report. (#5855). Contributed by Yanlei Yu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-28 14:52:38 +08:00