big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,657 Commits 2 Branches 2 Tags 264 MiB
e9740cb17a
Commit Graph

26657 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viraj Jasani
0e3aafe6c0
HADOOP-18399. S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator (#5054) 
Contributed by Viraj Jasani
 2023-04-18 16:37:48 +01:00
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
Steve Loughran
6ea10cf41b
HADOOP-18696. ITestS3ABucketExistence arn test failures. (#5557) 
Explicitly sets the fs.s3a.endpoint.region to eu-west-1 so
the ARN-referenced fs creation fails with unknown store
rather than IllegalArgumentException.

Steve Loughran
 2023-04-17 10:18:33 +01:00
yl09099
2c4d6bf33d
YARN-11465. Improved YarnClient Log Format (#5550) 
Co-authored-by: yl09099 <shaq376260428@163.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-17 09:27:52 +08:00
Dongjoon Hyun
0d1b4a3556
HADOOP-18590. Publish SBOM artifacts (#5555). Contributed by Dongjoon Hyun. 2023-04-15 21:35:43 +05:30
slfan1989
0bcdea7912
YARN-11239. Optimize FederationClientInterceptor audit log. (#5127) 2023-04-14 13:09:18 -07:00
zhangshuyan
0185afafea
HDFS-16974. Consider volumes average load of each DataNode when choosing target. (#5541). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-14 10:33:30 +08:00
dependabot[bot]
f1936d29f1
HADOOP-18693. Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project (#5427) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-13 10:25:17 -07:00
Melissa You
2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 (#5536) 2023-04-13 09:07:42 -07:00
slfan1989
06f9bdffa6
YARN-10846. Add dispatcher metrics to NM. (#4687) 2023-04-12 09:53:20 -07:00
slfan1989
dd6d0ac510
YARN-11462. Fix Typo of hadoop-yarn-common. (#5539) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-12 11:08:23 +08:00
Steve Loughran
7c3d94a032
HADOOP-18637. S3A to support upload of files greater than 2 GB using DiskBlocks (#5543) 
Contributed By: HarshitGupta and Steve Loughran
 2023-04-12 05:17:45 +05:30
slfan1989
bffa49a64f
YARN-11377. [Federation] Support addToClusterNodeLabels、removeFromClusterNodeLabels、replaceLabelsOnNode API's for Federation. (#5525) 2023-04-11 09:47:58 -07:00
Sadanand Shenoy
74ddf69f80
HDFS-16911. Distcp with snapshot diff to support Ozone filesystem. (#5364) 2023-04-10 14:03:16 -07:00
rdingankar
3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… (#5495) 2023-04-10 08:56:00 -07:00
mjwiq
e45451f9c7
HADOOP-18687. hadoop-auth: remove unnecessary dependency on json-smart (#5524) 
Contributed by Michiel de Jong
 2023-04-06 16:00:33 +01:00
zhtttylz
523ff81624
HDFS-16952. Support getLinkTarget API in WebHDFS (#5517) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-06 19:44:47 +08:00
Simbarashe Dzinamarira
47c22e388e
HDFS-16943. RBF: Implements MySQL based StateStoreDriver. (#5469) 2023-04-05 16:44:29 -07:00
Viraj Jasani
422bf3b24c
HDFS-16973. RBF: MountTableResolver cache size lookup should take read lock (#5533) 2023-04-05 14:06:38 -07:00
slfan1989
69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
HarshitGupta11
dfb2ca0a64
HADOOP-18684. S3A filesystem to support binding to to other URI schemes (#5521) 
Contributed by Harshit Gupta
 2023-04-05 12:42:11 +01:00
Viraj Jasani
937caf7de9
HDFS-16967. RBF: File based state stores should allow concurrent access to the records (#5523) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-04-04 22:39:53 +09:00
Chris Nauroth
14c5810d5e HADOOP-18680: Insufficient heap during full test runs in Docker container. 
Closes #5522

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-03 22:53:29 +00:00
zhtttylz
811441d5bc
HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc (#5491) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-01 18:18:20 +08:00
slfan1989
eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00
sreeb-msft
389b3ea6e3
HADOOP-18012. ABFS: Enable config controlled ETag check for Rename idempotency (#5488) 
To support recovery of network failures during rename, the abfs client
fetches the etag of the source file, and when recovering from a
failure, uses this tag to determine whether the rename succeeded
before the failure happened.

* This works for files, but not directories
* It adds the overhead of a HEAD request before each rename.
* The option can be disabled by setting "fs.azure.enable.rename.resilience"
  to false

Contributed by Sree Bhattacharyya
 2023-03-31 19:15:15 +01:00
Galsza
016362a28b
HADOOP-18548. Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems (#5355) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-03-30 07:12:02 +08:00
Viraj Jasani
b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
slfan1989
5bc8f25327
YARN-11446. [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. (#5476) 2023-03-28 09:33:19 -07:00
slfan1989
aa602381c5
YARN-11426. Improve YARN NodeLabel Memory Display. (#5335) 
YARN-11426. Improve YARN NodeLabel Memory Display.
Co-authored-by: slfan1989 <louj1988@@>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-03-28 22:48:46 +08:00
zhangshuyan
700147b4ac
HDFS-16964. Improve processing of excess redundancy after failover. (#5510). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-28 16:14:59 +08:00
Jinhu Wu
b5e8269d9b
HADOOP-18458: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS (#4912) 2023-03-28 14:27:01 +08:00
slfan1989
926993cb73
YARN-11376. [Federation] Support updateNodeResource、refreshNodesResources API's for Federation. (#5496) 2023-03-27 09:27:21 -07:00
Anmol Asrani
762d3ddb43
HADOOP-18146: ABFS: Added changes for expect hundred continue header (#4039) 
This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
 2023-03-27 12:43:34 +01:00
Andras Katona
ee01c64c6c
HADOOP-18676. jettison dependency override in hadoop-common lib (#5513) 2023-03-27 09:59:02 +02:00
Ayush Saxena
b82bcbd8ad
Revert "HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507)" 
This reverts commit 72b0122706.
 2023-03-25 12:04:28 +05:30
Andras Katona
72b0122706
HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507) 
* HADOOP-18587. Fixing jettison vulnerability of hadoop-common lib

* no need for excluding, let it come

Change-Id: Ia6e4ad351158dd4b0510dec34bbde531a60e7654
 2023-03-24 16:31:45 +01:00
Tamas Domok
69748aae32
YARN-11461. fix NPE in determineMissingParents (auto queue creation / CS). (#5506) 
Change-Id: Iaaaf43a545588eaff8a0a20f6f3c27258a45f390
 2023-03-24 09:38:53 +01:00
Kidd5368
5cf62d1498
HDFS-16948. Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. (#5474). Contributed by Kidd53685368. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 17:00:23 +08:00
zhaixiaojuan@loongson.cn
028cde0006
HADOOP-18644. Add bswap support for LoongArch64. (#5453). Contributed by zhaixiaojuan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 11:08:59 +08:00
Ayush Saxena
e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Yubi Lee
67e02a92e0
HADOOP-18666. A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server (#5480) 2023-03-22 10:54:41 +09:00
Viraj Jasani
0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem (#5492) 2023-03-21 10:23:16 +08:00
Viraj Jasani
9a8287c36f
HADOOP-18669. Remove Log4Json Layout (#5493) 2023-03-21 10:07:06 +08:00
Viraj Jasani
f8d0949f7d
HDFS-16953. RBF: Mount table store APIs should update cache only if state store record is successfully updated (#5482) 2023-03-18 14:43:25 -07:00
Viraj Jasani
b6a9d7b442
HADOOP-18631. (ADDENDUM) Use LogCapturer to match audit log pattern and remove hdfs async audit log configs (#5451) 2023-03-18 06:33:50 +08:00
slfan1989
fa723ae839
YARN-11445. [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. (#5472) 2023-03-17 11:59:45 -07:00
Pranav Saxena
759ddebb13
HADOOP-18647. x-ms-client-request-id to identify the retry of an API. (#5437) 
The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous
operation

Contributed by Pranav Saxena
 2023-03-15 20:03:22 +00:00
Masatake Iwasaki
7c42d0f7da
HADOOP-17746. Compatibility table in directory_markers.md doesn't render right. (#3116) 
Contributed by Masatake Iwasaki
 2023-03-15 17:10:42 +00:00
Viraj Jasani
15935fa865
HDFS-16947. RBF NamenodeHeartbeatService to report error for not being able to register namenode in state store (#5470) 2023-03-16 00:59:55 +08:00