big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,271 Commits 2 Branches 2 Tags 264 MiB
700b3e4800
Commit Graph

27271 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Anuj Modi
e3c135b0b3
HADOOP-18971. [ABFS] Read and cache file footer with fs.azure.footer.read.request.size (#6270) 
The option fs.azure.footer.read.request.size sets the size of the footer to
read and cache; the default value of 524288 has been measured to
be good for most workloads running on parquet, ORC and similar file formats.

Contributed by Anuj Modi
 2024-01-03 12:49:52 +00:00
slfan1989
556fbcf025
YARN-11632. [Doc] Add allow-partial-result description to Yarn Federation documentation. (#6340) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-03 07:17:37 +08:00
Pranav Saxena
0b43026cab
HADOOP-17912. ABFS: Support for Encryption Context (#6221) 
Contributed by Pranav Saxena and others.
 2024-01-01 19:09:44 +00:00
Murali Krishna
9edcf42c78
HADOOP-18540. Upgrade Bouncy Castle to 1.70 (#5166) 
This addresses
- [sonatype-2021-4916] CWE-327: Use of a Broken or Risky Cryptographic Algorithm
- [sonatype-2019-0673] CWE-400: Uncontrolled Resource Consumption ('Resource Exhaustion')

Contributed by Murali Krishna
 2024-01-01 19:04:06 +00:00
Ayush Saxena
9a4d10763c
HADOOP-19020. Update the year to 2024. (#6397). Contributed by Ayush Saxena. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2024-01-01 12:51:54 +05:30
zzccctv
9f76fba6a4
Delete invalid code logic in namenode format (#6323) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 22:25:05 +09:00
Gautham B A
98656db736
HADOOP-19017. Setup pre-commit CI for Windows 10 (#5820) 
* This PR adds a Jenkinsfile for pre-commit CI
  to validate the Hadoop PRs on Windows 10.
 2023-12-29 16:41:10 +05:30
huangzhaobo
e26139beaa
HDFS-17301. Add read and write dataXceiver threads count metrics to datanode. (#6377) 
Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 12:43:46 +09:00
huhaiyang
3aceec711b
HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted (#6369) 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-28 22:13:16 +09:00
zhtttylz
fc71dc3e94
HDFS-17284. Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery (#6348) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-27 10:43:10 +09:00
Lei313
773dd7cc85
HDFS-17275: judge whether the block has been deleted in the block report (#6335). Contributed by lei w. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-26 18:25:02 +08:00
huhaiyang
7e2ebfc17e
HDFS-17298. Fix NPE in DataNode.handleBadBlock and BlockSender (#6374) 2023-12-26 09:41:10 +09:00
xiaojunxiang
e07e445326
HDFS-17215. RBF: Fix some method annotations about @throws . (#6136). Contributed by xiaojunxiang. 2023-12-26 00:37:58 +05:30
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. (#6379). Contributed by huangzhaobo99. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-25 23:17:25 +05:30
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging (#6321). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-24 16:57:04 +08:00
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java (#6226). Contributed by Yu Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2023-12-22 01:52:12 +05:30
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration (#6347) Contributed by LiuGuH. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-21 12:21:59 +08:00
K0K0V0K
5dd1977800
YARN-11634. Speed-up TestTimelineClient (#6371) 
The TimelineConnector.class has a hardcoded 1 minute connection time out, what makes the TestTimelineClient to a long running test (~15:30 min).
Decrease the timeout to 10ms will speed up the test run (~56 sec).
 2023-12-20 12:10:07 +01:00
Mukund Thakur
01bde4afff Revert "HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool" 
Pushed it by mistake. So sorry.
This reverts commit e28f83a1eb.
 2023-12-19 14:12:21 -06:00
Mukund Thakur
e28f83a1eb HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool 2023-12-19 14:04:07 -06:00
BilwaST
f52c7d3e9a
HADOOP-18613. Upgrade ZooKeeper to version 3.8.3 (#6296). Contributed by Bilwa S T. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 23:01:28 +05:30
gavin.wang
de9fde8224
YARN-11563. Fix typo in AbstractContainerAllocator from CSAssignemnt to CSAssignment (#6026). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:43:56 +05:30
gavin.wang
b5f68b6225
MAPREDUCE-7463. Fix missing comma in HistoryServerRest.html response body (#6342). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:40:55 +05:30
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. (#6366) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-19 09:32:47 +09:00
Anika Kelhanka
62cc673d00
[HADOOP-19010] - NullPointerException in Hadoop Credential Check CLI (#6351) 2023-12-16 12:23:52 +05:30
slfan1989
3caabb2957
YARN-11629. [GPG] Improve GPGOverviewBlock Infomation. (#6341) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 12:36:57 +08:00
slfan1989
7935eded5e
YARN-11627. [GPG] Improve GPGPolicyFacade#getPolicyManager. (#6332) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 08:36:11 +08:00
Doroszlai, Attila
cf21f35526
HADOOP-19011. Possible ConcurrentModificationException if Exec command fails (#6353) 2023-12-14 19:46:19 +01:00
Peter Szucs
16edb4ac96
YARN-11630. Passing admin Java options to container localizers (#6349) 2023-12-14 16:52:57 +01:00
huangzhaobo
1498a8685d
HDFS-17282. Reconfig 'SlowIoWarningThreshold' parameters for datanode. (#6338). Contributed by huangzhaobo99 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-14 11:47:53 +05:30
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. (#6227) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-12-13 20:56:53 +08:00
Steve Loughran
19b9e6a97b
HADOOP-19008. S3A: update aws-sdk version to 2.21.41 (#6334) 
AWS SDK is now at 2.21.41.
Key change: log4j.properties settings are picked up.
 2023-12-12 15:15:32 +00:00
yijut2
81de229cf6
HDFS-17278. Fix order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module (#6329). Contributed by Ruby. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-12 12:42:26 +05:30
hfutatzhanghb
e91daae318
HDFS-17152. Fix the documentation of count command in FileSystemShell.md. (#5939). Contributed by farmmamba. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-11 16:53:37 +08:00
caozhiqiang
37d6cada14
HDFS-17272. NNThroughputBenchmark should support specifying the base directory for multi-client test (#6319). Contributed by caozhiqiang. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-10 13:43:04 +05:30
huhaiyang
f96361fd54
HDFS-17279. RBF: Fix link to Fedbalance document (#6333). Contributed by Haiyang Hu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-09 02:32:04 +05:30
Jian Zhang
29b0a6c28d
HDFS-17265. RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController (#6298) 2023-12-08 10:18:22 -08:00
hfutatzhanghb
e91aec930f
HADOOP-18989. Use thread pool to improve the speed of creating control files in TestDFSIO (#6294). Contributed by farmmamba. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-08 17:15:58 +08:00
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
Steve Loughran
25089dc9ee
HADOOP-18997. S3A: Add option fs.s3a.s3express.create.session to disable S3Express CreateSession (#6316) 
Adds a new option fs.s3a.s3express.create.session; default is true.

When false, this disables the CreateSession call to create/refresh temporary
session credentials when working with an Amazon S3 Express store.

This avoids having to give the caller the new IAM role permission,
at the expense of every remote call on the S3 Express store having to
include the latency of a checkup of IAM permissions.

* fs.s3a.s3express.create.session is set to false in tests which generate new
  role permissions and call AssumeRole
* move ApiCallTimeoutException logic until after sdk exceptions get translated
  to IOE. This lines up for any future AWS throwing up underlying cause here.
* Tests will automatically skip ACL, storage class, S3 Select or encryption tests
when target fs is S3Express.
* same for the out of order multipart uploader test cases, v1 listing.
* bucket tool s3 test treats invalid location error as a successful invocation
  of the create bucket attempt

Contributed by Steve Loughran
 2023-12-07 13:08:47 +00:00
Xing Lin
607c981042
HDFS-17262. Fixed the verbose log.warn in DFSUtil.addTransferRateMetric(). (#6290). Contributed by Xing Lin. 
Reviewed-by: Ravindra Dingankar <rdingankar@linkedin.com>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-07 10:16:23 +08:00
Thinker313
6a22bead54
HDFS-17270. RBF: Fix ZKDelegationTokenSecretManagerImpl use closed zookeeper client to get token in some case (#6315). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-06 21:34:46 +08:00
zhangshuyan
809ae58e71
HADOOP-18982. Fix doc about loading native libraries. (#6281). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-06 21:24:14 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
K0K0V0K
5d1f889432
YARN-11420. Stabilize TestNMClient (#5317) 2023-12-05 11:32:42 +01:00
Susheel Gupta
78d5fe24d6
YARN-11621: Fix intermittently failing unit test: TestAMRMProxy.testAMRMProxyTokenRenewal (#6310) 2023-12-05 11:26:46 +01:00
huhaiyang
1fb80ef787
HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted (#6266). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:19:33 +08:00
huhaiyang
9a6d00aba4
HDFS-17218. NameNode should process time out excess redundancy blocks (#6176). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:16:38 +08:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
huangzhaobo
2323ad24a2
HDFS-17260. Fix the logic for reconfigure slow peer enable for Namenode. (#6279). Contributed by huangzhaobo99. 
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-03 15:09:55 +05:30