big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,190 Commits 2 Branches 2 Tags 264 MiB
77c600d769
Commit Graph

27190 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. (#6357). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:34:02 +08:00
Steve Loughran
2f1e1558b6
HADOOP-19004. S3A: Support Authentication through HttpSigner API (#6324) 
Move to the new auth flow based signers for aws. * Implement a new Signer Initialization Chain
* Add a new instantiation method
* Add a new test
* Fix Reflection Code for SignerInitialization

Contributed by Harshit Gupta
 2024-01-11 17:13:31 +00:00
Xing Lin
453e264eb4
HADOOP-18981. Move oncrpc and portmap packages to hadoop-common (#6280) 
Move the org.apache.hadoop.{oncrpc, portmap} packages from the hadoop-nfs module
to the hadoop-common module.

This allows for use of the protocol beyond just NFS -including within HDFS itself.

Contributed by Xing Lin
 2024-01-11 14:06:15 +00:00
Benjamin Teke
ef636c4278
MAPREDUCE-7468: Change add-opens flag's default value from true to false (#6436) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2024-01-11 14:51:59 +01:00
hfutatzhanghb
6a053765ee
HDFS-17312. packetsReceived metric should ignore heartbeat packet. (#6394) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-11 22:08:37 +09:00
Tamas Domok
55b9f87698
YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. (#6433) 2024-01-11 13:47:00 +01:00
slfan1989
bc159b5a87
YARN-10125. [Federation] Kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM. (#6363) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-11 20:01:59 +08:00
xuzifu666
99a59ae9e6
HDFS-17317. Improve the resource release for metaOut in DebugAdmin (#6402). Contributed by xy. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2024-01-07 00:59:31 +05:30
slfan1989
64beecb7cb
YARN-11631. [GPG] Add GPGWebServices. (#6354) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-06 17:50:20 +08:00
slfan1989
60033fd581
YARN-11642. Fix Flaky Test TestTimelineAuthFilterForV2#testPutTimelineEntities. (#6417) Contributed by Shilun Fan. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-06 16:26:01 +08:00
huangzhaobo
08713665c0
HDFS-17315. Optimize the namenode format code logic. (#6400) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-06 01:47:17 +09:00
LiuGuH
5f9932acc4
HDFS-17325. Fix the documentation of fs expunge command in FileSystemShell.md. (#6413) Contributed by liuguanghua. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-05 18:42:55 +08:00
LiuGuH
2369f0cddb
HDFS-17309. RBF: Fix Router Safemode check condition error (#6390) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-05 18:36:28 +08:00
Lei Yang
661c784662
HDFS-17290: Adds disconnected client rpc backoff metrics (#6359) 2024-01-04 20:24:10 -08:00
LiuGuH
7d3b6a36b8
HDFS-17306. RBF: Router should not return nameservices that does not enable observer nodes in RpcResponseHeaderProto (#6385) 2024-01-04 14:43:11 -08:00
hfutatzhanghb
8c26d4e9e0
HDFS-17322. Renames RetryCache#MAX_CAPACITY to be MIN_CAPACITY to fit usage. 2024-01-04 14:31:53 -08:00
hfutatzhanghb
d5468d84ba
HDFS-17283. Change the name of variable SECOND in HdfsClientConfigKeys. (#6339). Contributed by farmmamba. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-04 19:53:47 +08:00
huhaiyang
7a7db7f0dc
HDFS-17310. DiskBalancer: Enhance the log message for submitPlan (#6391) Contributed by Haiyang Hu. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-04 00:07:51 +08:00
LiuGuH
335587df9e
Add synchronized on lockLeakCheck() because threadCountMap is not thread safe. (#6029) 
Co-authored-by: lgh <liuguanghua@kanzhun.com>
 2024-01-03 21:12:38 +08:00
Anuj Modi
e3c135b0b3
HADOOP-18971. [ABFS] Read and cache file footer with fs.azure.footer.read.request.size (#6270) 
The option fs.azure.footer.read.request.size sets the size of the footer to
read and cache; the default value of 524288 has been measured to
be good for most workloads running on parquet, ORC and similar file formats.

Contributed by Anuj Modi
 2024-01-03 12:49:52 +00:00
slfan1989
556fbcf025
YARN-11632. [Doc] Add allow-partial-result description to Yarn Federation documentation. (#6340) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-03 07:17:37 +08:00
Pranav Saxena
0b43026cab
HADOOP-17912. ABFS: Support for Encryption Context (#6221) 
Contributed by Pranav Saxena and others.
 2024-01-01 19:09:44 +00:00
Murali Krishna
9edcf42c78
HADOOP-18540. Upgrade Bouncy Castle to 1.70 (#5166) 
This addresses
- [sonatype-2021-4916] CWE-327: Use of a Broken or Risky Cryptographic Algorithm
- [sonatype-2019-0673] CWE-400: Uncontrolled Resource Consumption ('Resource Exhaustion')

Contributed by Murali Krishna
 2024-01-01 19:04:06 +00:00
Ayush Saxena
9a4d10763c
HADOOP-19020. Update the year to 2024. (#6397). Contributed by Ayush Saxena. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2024-01-01 12:51:54 +05:30
zzccctv
9f76fba6a4
Delete invalid code logic in namenode format (#6323) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 22:25:05 +09:00
Gautham B A
98656db736
HADOOP-19017. Setup pre-commit CI for Windows 10 (#5820) 
* This PR adds a Jenkinsfile for pre-commit CI
  to validate the Hadoop PRs on Windows 10.
 2023-12-29 16:41:10 +05:30
huangzhaobo
e26139beaa
HDFS-17301. Add read and write dataXceiver threads count metrics to datanode. (#6377) 
Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 12:43:46 +09:00
huhaiyang
3aceec711b
HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted (#6369) 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-28 22:13:16 +09:00
zhtttylz
fc71dc3e94
HDFS-17284. Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery (#6348) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-27 10:43:10 +09:00
Lei313
773dd7cc85
HDFS-17275: judge whether the block has been deleted in the block report (#6335). Contributed by lei w. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-26 18:25:02 +08:00
huhaiyang
7e2ebfc17e
HDFS-17298. Fix NPE in DataNode.handleBadBlock and BlockSender (#6374) 2023-12-26 09:41:10 +09:00
xiaojunxiang
e07e445326
HDFS-17215. RBF: Fix some method annotations about @throws . (#6136). Contributed by xiaojunxiang. 2023-12-26 00:37:58 +05:30
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. (#6379). Contributed by huangzhaobo99. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-25 23:17:25 +05:30
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging (#6321). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-24 16:57:04 +08:00
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java (#6226). Contributed by Yu Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2023-12-22 01:52:12 +05:30
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration (#6347) Contributed by LiuGuH. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-21 12:21:59 +08:00
K0K0V0K
5dd1977800
YARN-11634. Speed-up TestTimelineClient (#6371) 
The TimelineConnector.class has a hardcoded 1 minute connection time out, what makes the TestTimelineClient to a long running test (~15:30 min).
Decrease the timeout to 10ms will speed up the test run (~56 sec).
 2023-12-20 12:10:07 +01:00
Mukund Thakur
01bde4afff Revert "HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool" 
Pushed it by mistake. So sorry.
This reverts commit e28f83a1eb.
 2023-12-19 14:12:21 -06:00
Mukund Thakur
e28f83a1eb HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool 2023-12-19 14:04:07 -06:00
BilwaST
f52c7d3e9a
HADOOP-18613. Upgrade ZooKeeper to version 3.8.3 (#6296). Contributed by Bilwa S T. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 23:01:28 +05:30
gavin.wang
de9fde8224
YARN-11563. Fix typo in AbstractContainerAllocator from CSAssignemnt to CSAssignment (#6026). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:43:56 +05:30
gavin.wang
b5f68b6225
MAPREDUCE-7463. Fix missing comma in HistoryServerRest.html response body (#6342). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:40:55 +05:30
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. (#6366) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-19 09:32:47 +09:00
Anika Kelhanka
62cc673d00
[HADOOP-19010] - NullPointerException in Hadoop Credential Check CLI (#6351) 2023-12-16 12:23:52 +05:30
slfan1989
3caabb2957
YARN-11629. [GPG] Improve GPGOverviewBlock Infomation. (#6341) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 12:36:57 +08:00
slfan1989
7935eded5e
YARN-11627. [GPG] Improve GPGPolicyFacade#getPolicyManager. (#6332) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 08:36:11 +08:00
Doroszlai, Attila
cf21f35526
HADOOP-19011. Possible ConcurrentModificationException if Exec command fails (#6353) 2023-12-14 19:46:19 +01:00
Peter Szucs
16edb4ac96
YARN-11630. Passing admin Java options to container localizers (#6349) 2023-12-14 16:52:57 +01:00
huangzhaobo
1498a8685d
HDFS-17282. Reconfig 'SlowIoWarningThreshold' parameters for datanode. (#6338). Contributed by huangzhaobo99 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-14 11:47:53 +05:30
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. (#6227) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-12-13 20:56:53 +08:00