big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,395 Commits 2 Branches 2 Tags 264 MiB
0837c84a9f
Commit Graph

27395 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Anuj Modi
000a39ba2d
HADOOP-18872: [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations (#6019) 
Contributed by Anuj Modi
 2023-11-13 19:36:33 +00:00
LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure (#6249) 
Contributed by liuguanghua.
 2023-11-13 18:16:31 +00:00
K0K0V0K
a32097a921
HADOOP-18954. Filter NaN values from JMX json interface. (#6229). 
Reviewed-by: Ferenc Erdelyi
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-11-09 17:14:14 +08:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (#6198) 
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
 2023-11-08 14:25:21 +00:00
slfan1989
90e9aa272e
YARN-11484. [Federation] Router Supports Yarn Client CLI Cmds. (#6132) 2023-11-07 18:38:35 -08:00
slfan1989
d9a6792ca9
YARN-11011. Make YARN Router throw Exception to client clearly. (#6211) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 10:07:54 +08:00
slfan1989
72d7b43a32
YARN-11548. [Federation] Router Supports Format FederationStateStore. (#6116) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 07:41:56 +08:00
Anuj Modi
597ceaae3a
HADOOP-18874: [ABFS] Add Server request ID in Exception Messages thrown to the caller. (#6004) 
Contributed by Anuj Modi
 2023-11-06 20:56:55 +00:00
Steve Loughran
d634deea4e
HADOOP-18487. Protobuf 2.5 removal part 2: stop exporting protobuf-2.5 (#6185) 
Followup to the previous HADOOP-18487 patch: changes the scope of
protobuf-2.5 in hadoop-common and elsewhere from "compile" to "provided".

This means that protobuf-2.5 is
* No longer included in hadoop distributions
* No longer exported by hadoop common POM files
* No longer exported transitively by other hadoop modules.
* No longer listed in LICENSE-binary.

Contributed by Steve Loughran
 2023-11-06 17:52:05 +00:00
Steve Loughran
ef7fb64764
HADOOP-18925. S3A: option to enable/disable CopyFromLocalOperation (#6163) 
Add a new option:
fs.s3a.optimized.copy.from.local.enabled

This will enable (default) or disable the
optimized CopyFromLocalOperation upload operation
when copyFromLocalFile() is invoked.

When false the superclass implementation is used; duration
statistics are still collected, though audit span entries
in logs will be for the individual fs operations, not the
overall operation.

Contributed by Steve Loughran
 2023-11-06 16:00:56 +00:00
Benjamin Teke
077263d9f3
YARN-11608. Fix QueueCapacityVectorInfo NPE when accessible labels config is used. (#6250) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-06 10:11:14 +01:00
huhaiyang
4ef2322b6d
HDFS-17243. Add the parameter storage type for getBlocks method (#6238). Contributed by Haiyang Hu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-11-06 11:20:25 +08:00
Junfan Zhang
c15fd3b2c0
YARN-11599. Incorrect log4j properties file in SLS sample conf (#6220) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-05 13:57:48 +08:00
WangYuanben
a79cd110a4
YARN-11609. [Addendum] [Federation] Router Supports DeregisterSubCluster. (#6254) YuanBen Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-05 07:50:41 +08:00
slfan1989
0a867b9c39
YARN-11594. [Federation] Improve Yarn Federation documentation. (#6209) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-04 19:09:48 +08:00
rRajivramachandran
d6bb47e5db
MAPREDUCE-7459. Fixed TestHistoryViewerPrinter flakiness during string comparison (#6215). Contributed by Rajiv Ramachandran. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-04 07:23:50 +05:30
YuanHanzhong
5b215f23d5
HADOOP-18963. Fix typos in .gitignore (#6243) 2023-11-04 05:12:39 +05:30
slfan1989
2b1378c99e
YARN-11583. [Addendum] Improve Node Link for YARN Federation Web Page. (#6232) 2023-11-01 11:51:38 -07:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows (#5943) 2023-11-01 09:10:15 -07:00
Viraj Jasani
cf3a4b3bb7
HADOOP-18850. S3A: Enable dual-layer server-side encryption with AWS KMS keys (#6140) 
Contributed by Viraj Jasani
 2023-11-01 13:30:35 +00:00
Brian Goerlitz
4c04a6768c
YARN-11584. Safely fail on leaf queue with empty name (#6148) 2023-10-31 17:25:41 +01:00
mudit1289
f1ce273150
MAPREDUCE-7457: Added support to limit count of spill files (#6155) Contributed by Mudit Sharma. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-31 06:58:22 +08:00
slfan1989
254dbab5a3
YARN-9013. [BackPort] [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner. (#6147) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-31 06:56:00 +08:00
PJ Fanning
a079f6261d
HADOOP-18917. Addendum. Fix deprecation issues after commons-io upgrade. (#6228). Contributed by PJ Fanning. 2023-10-30 09:35:02 +05:30
ConfX
7c6af6a5f6
HADOOP-18905. Negative timeout in ZKFailovercontroller due to overflow. (#6092). Contributed by ConfX. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-29 13:30:28 +05:30
PJ Fanning
b9c9c42b29
HADOOP-18936. Upgrade to jetty 9.4.53 (#6181). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-29 13:09:12 +05:30
slfan1989
40e8300719
YARN-11592. Add timeout to GPGUtils#invokeRMWebService. (#6189) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-28 07:09:09 +08:00
Junfan Zhang
e4eda40ac9
YARN-11597. Fix NPE when loading static files in SLSWebApp (#6216) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-27 22:11:01 +08:00
Steve Loughran
7ec636deec
HADOOP-18930. Make fs.s3a.create.performance a bucket-wide setting. (#6168) 
If fs.s3a.create.performance is set on a bucket
- All file overwrite checks are skipped, even if the caller says
  otherwise.
- All directory existence checks are skipped.
- Marker deletion is *always* skipped.

This eliminates a HEAD and a LIST for every creation.

* New path capability "fs.s3a.create.performance.enabled" true
  if the option is enabled.
* Parameterize ITestS3AContractCreate to expect the different
  outcomes
* Parameterize ITestCreateFileCost similarly, with
  changed cost assertions there.
* create(/) raises an IOE. existing bug only noticed here.

Contributed by Steve Loughran
 2023-10-27 12:23:55 +01:00
Hiroaki Segawa
93a3c6e2cd
HDFS-17024. Potential data race introduced by HDFS-15865 (#6223) 2023-10-26 22:25:00 -07:00
slfan1989
652908519e
YARN-11588. [Federation] [Addendum] Fix uncleaned threads in yarn router thread pool executor. (#6222) 2023-10-26 13:39:06 -07:00
Wei-Chiu Chuang
821ed83873
HDFS-15273. CacheReplicationMonitor hold lock for long time and lead to NN out of service. Contributed by Xiaoqiao He. 2023-10-26 10:35:10 -07:00
slfan1989
d18410221b
YARN-11593. [Federation] Improve command line help information. (#6199) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-26 08:22:18 +08:00
Steve Loughran
8bd1f65efc
HADOOP-18948. S3A. Add option fs.s3a.directory.operations.purge.uploads to purge on rename/delete (#6218) 
S3A directory delete and rename will optionally abort all pending multipart uploads
in their under their to-be-deleted paths when.

fs.s3a.directory.operations.purge.upload is true

It is off by default.

The filesystems hasPathCapability("fs.s3a.directory.operations.purge.upload")
probe will return true when this feature is enabled.

Multipart uploads may accrue from interrupted data writes, uncommitted 
staging/magic committer jobs and other operations/applications. On AWS S3
lifecycle rules are the recommended way to clean these; this change improves
support for stores which lack these rules.

Contributed by Steve Loughran
 2023-10-25 17:39:16 +01:00
PJ Fanning
bbf905dc99
HADOOP-18933. upgrade to netty 4.1.100 due to CVE (#6173) 
Mitigates Netty security advisory GHSA-xpw8-rcwv-8f8p
"HTTP/2 Rapid Reset Attack - DDoS vector in the HTTP/2 protocol due RST frames"

Contributed by PJ Fanning
 2023-10-25 14:06:13 +01:00
huhaiyang
f85ac5b60d
HADOOP-18920. RPC Metrics : Optimize logic for log slow RPCs (#6146) 2023-10-25 13:56:39 +08:00
gp1314
a170d58501
HDFS-17231. HA: Safemode should exit when resources are from low to available. (#6207). Contributed by Gu Peng. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-10-25 11:43:12 +08:00
Stephen O'Donnell
882f08b4bc
HDFS-17237. Remove IPCLoggerChannelMetrics when the logger is closed (#6217) 2023-10-24 21:39:03 +01:00
Steve Loughran
8b974bcc1f
HADOOP-18889. Third party storage followup. (#6186) 
Followup to HADOOP-18889 third party store support;

Fix some minor review comments which came in after the merge.
 2023-10-24 18:17:52 +01:00
PJ Fanning
0042544bf2
HADOOP-18949. upgrade maven dependency plugin due to CVE-2021-26291. (#6219) 
Addresses CVE-2021-26291. "Origin Validation Error in Apache Maven"

Contributed by PJ Fanning.
 2023-10-24 12:28:40 +01:00
slfan1989
9c7e5b66fa
YARN-11576. Improve FederationInterceptorREST AuditLog. (#6117) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-24 09:36:06 +08:00
slfan1989
80a22a736e
YARN-11500. [Addendum] Fix typos in hadoop-yarn-server-common#federation. (#6212) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-24 09:28:05 +08:00
huhaiyang
9d48af8d70
HADOOP-18868. Optimize the configuration and use of callqueue overflow trigger failover (#5998) 2023-10-23 14:06:02 -07:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
huhaiyang
5eeab5e1b9
HDFS-17235. Fix javadoc errors in BlockManager (#6214). Contributed by Haiyang Hu. 2023-10-23 20:12:39 +05:30
jianghuazhu
6e13e4addc
HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-23 20:05:33 +05:30
Ayush Saxena
fbd653be9b
Revert "HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu." 
This reverts commit 81ba2e8484.
 2023-10-23 19:35:12 +05:30
Steve Loughran
3e0fcda7a5
HADOOP-18945. S3A. IAMInstanceCredentialsProvider failing. (#6202) 
This restores asynchronous retrieval/refresh of any AWS credentials provided by the
EC2 instance/container in which the process is running.

Contributed by Steve Loughran
 2023-10-23 14:24:30 +01:00
slfan1989
d7d772d684
YARN-11595. Fix hadoop-yarn-client#java.lang.NoClassDefFoundError (#6210) Contributed by Shilun Fan. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-22 22:22:14 +08:00
Masatake Iwasaki
24fe1ef4dd HADOOP-18942 addendum. update LICENSE-binary. 2023-10-22 22:22:56 +09:00