big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,323 Commits 2 Branches 2 Tags 264 MiB
a1f5dc5865
Commit Graph

27323 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
0d057e27c3
MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378) 
By default, the mapreduce manifest committer is used for jobs working with abfs and gcs.
Hadoop mapreduce will pick this up automatically; for Spark it is a bit complicated: read the docs
to see the steps required.
 2023-06-27 13:55:20 +01:00
PJ Fanning
56ef05a9ca
HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs (#5773) 
Addresses CVE-2023-34454

Contributed by PJ Fanning
 2023-06-27 11:53:02 +01:00
rohit-kb
21d9c4727c
HADOOP-18773. Upgrade maven-shade-plugin to 3.4.1 (#5750) 
This is needed to successfully shade the hadoop binaries on recent Java versions.

Contributed by Rohit Kumar
 2023-06-27 10:40:22 +01:00
Xing Lin
03902f5ef0
HDFS-17055 Export HAState as a metric from Namenode for monitoring (#5764) 2023-06-26 15:52:59 -07:00
Gautham B A
a85272c33d
HADOOP-18751. Fix incorrect output path in javadoc build phase (#5688) 2023-06-26 15:52:17 -07:00
slfan1989
a4cf4c3778
YARN-11517. Improve Federation#RouterCLI DeregisterSubCluster Code. (#5766) 2023-06-26 09:43:49 -07:00
huhaiyang
161f80810f
HDFS-17057. Add DataNode maintenance states to Federation UI (#5775) 2023-06-26 09:38:57 -07:00
Wei-Chiu Chuang
e239d40ab1 Post release update 
* Add jdiff xml files from 3.3.6 release.
* Declare 3.3.6 as the latest stable release.
* Copy release notes.

(cherry picked from commit 7db9895000)
(cherry picked from commit cc121e2124aa01458dc296a060edc5e21a295268)
 2023-06-26 16:08:24 +00:00
Szilard Nemeth
1c15987ee3 MAPREDUCE-7441. Fix race condition in closing FadvisedFileRegion. Contributed by Benjamin Teke 2023-06-23 14:40:03 -04:00
Devaspati
eb88b9ff21
YARN-11498. Exclude jettison from jersey-json artifact as on older version is being pulled (#5623) 2023-06-23 16:45:51 +02:00
susheel-gupta
c82ea52e4c
YARN-11513: Applications submitted to ambiguous queue fail during recovery if "Specified" Placement Rule is used (#5748) 2023-06-23 13:09:23 +02:00
liangxs
cebcb44d37
HADOOP-18713. Update solr from 8.8.2 to 8.11.2 (#5459). Contributed by Xuesen Liang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-22 13:15:57 +05:30
K0K0V0K
3f4de919bd
YARN-11511. Improve TestRMWebServices test config and data. (#5745) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2023-06-21 14:15:11 +02:00
slfan1989
5c02f21f2e
YARN-11510. [Federation] Fix NodeManager#TestFederationInterceptor Flaky Unit Test. (#5733) 2023-06-20 10:25:46 -07:00
slfan1989
3d2cddd235
YARN-11516. Improve FederationStateStoreFacade#existsApplicationHomeSubCluster/existsReservationHomeSubCluster Log Level. (#5755) 2023-06-20 10:23:51 -07:00
slfan1989
35d77a6359
YARN-11505. [Federation] Add Steps To Set up a Test Cluster. (#5751) 2023-06-20 10:23:11 -07:00
hfutatzhanghb
e14c52c25f
HDFS-17053. Optimize method BlockInfoStriped#findSlot to reduce time complexity. (#5757). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-19 14:49:23 +05:30
yl09099
7c54a78343
YARN-11506. The formatted yarn queue list is displayed on CLI (#5716). Contributed by Lu Yuan. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-18 14:39:25 +05:30
Hexiaoqiao
ba08f26a15
HADOOP-18774. Add .vscode to gitignore. (#5756). Contributed by Xiaoqiao He. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-18 14:08:38 +08:00
zhtttylz
9a7d1b49e2
HDFS-17043. HttpFS implementation for getAllErasureCodingPolicies (#5734). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-16 22:36:47 +05:30
Xing Lin
427366b73b
HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) 2023-06-15 13:59:58 -07:00
wangzhaohui
02027c8dcc
HDFS-17051. Fix wrong time unit in TestFileAppend4#recoverFile (#5749). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 18:20:53 +05:30
Viraj Jasani
2fe3b2a73f
HADOOP-18763. Upgrade aws-java-sdk to 1.12.367 (#5741) 
Contributed By: Viraj Jasani
 2023-06-15 01:09:41 +05:30
Tsz-Wo Nicholas Sze
5dd37b2814
HDFS-17045. File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted. (#5738) 2023-06-15 03:30:04 +08:00
hfutatzhanghb
1a918a6b96
HDFS-17047. BlockManager#addStoredBlock should log storage id when AddBlockResult is REPLACED (#5742). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 00:24:22 +05:30
Xing Lin
ab47a658bd
HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy (#5700) 2023-06-14 10:53:56 -07:00
yl09099
f4accec379
YARN-11504. [Addendum] [Federation] YARN Federation Supports Non-HA mode. (#5746) 2023-06-14 23:40:31 +08:00
Tamas Domok
f4b70b353d
YARN-11429. Improve the updateTestDataAutomatically in TestRMWebServices. (#5736) 2023-06-14 14:39:07 +02:00
Viraj Jasani
a75e378868
HADOOP-18756. S3A prefetch - CachingBlockManager to use AtomicBoolean for closed flag (#5718) 
Contributed by Viraj Jasani
 2023-06-14 12:51:54 +01:00
Viraj Jasani
f0c4286e3e
HDFS-17041. RBF: Fix putAll impl for mysql and file based state stores (#5723) 2023-06-13 09:55:04 -07:00
NishthaShah
3fbadc5d50
HDFS-16946. Fix getTopTokenRealOwners to return String (#5696). Contributed by Nishtha Shah. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-13 10:01:35 +05:30
Wei-Chiu Chuang
e1bb4acd36
HADOOP-18761. Remove mysql-connector-java (#5731) 2023-06-12 15:31:03 -07:00
slfan1989
a4c3d48c31
YARN-11504. [Federation] YARN Federation Supports Non-HA mode. (#5722) 2023-06-12 15:28:13 -07:00
slfan1989
2794fe264b
YARN-8898. [Addendum] Improve NodeManager#TestFederationInterceptor Setup Code (#5732) 2023-06-12 15:21:24 -07:00
slfan1989
a409d52ef8
YARN-6648. BackPort [GPG] Add SubClusterCleaner in Global Policy Generator. (#5676) 2023-06-12 15:18:43 -07:00
Dongjoon Hyun
fb16e00da0
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. 
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-11 11:38:13 +05:30
hfutatzhanghb
35158db711
HDFS-17023. RBF: Record proxy time when call invokeConcurrent method. (#5683). Contributed by farmmamba. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-10 00:06:28 +05:30
Steve Loughran
7a45ef4164
MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519) 
This modifies the manifest committer so that the list of files
to rename is passed between stages as a file of
writeable entries on the local filesystem.

The map of directories to create is still passed in memory;
this map is built across all tasks, so even if many tasks
created files, if they all write into the same set of directories
the memory needed is O(directories) with the
task count not a factor.

The _SUCCESS file reports on heap size through gauges.
This should give a warning if there are problems.

Contributed by Steve Loughran
 2023-06-09 17:00:59 +01:00
zhangshuyan
9c989515ba
HDFS-17037. Consider nonDfsUsed when running balancer. (#5715). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-09 16:19:08 +08:00
Steve Loughran
7bb09f1010
HADOOP-18752. Change fs.s3a.directory.marker.retention to "keep" (#5689) 
This 
1. changes the default value of fs.s3a.directory.marker.retention
   to "keep"
2. no longer prints a message when an S3A FS instances is
   instantiated with any option other than delete.

Switching to marker retention improves performance
on any S3 bucket as there are no needless marker DELETE requests
-leading to a reduction in write IOPS and and any delays waiting
for the DELETE call to finish.

There are *very* significant improvements on versioned buckets,
where tombstone markers slow down LIST operations: the more
tombstones there are, the worse query planning gets.

Having versioning enabled on production stores is the foundation
of any data protection strategy, so this has tangible benefits
in production.

It is *not* compatible with older hadoop releases; specifically
- Hadoop branch 2 < 2.10.2
- Any release of Hadoop 3.0.x and Hadoop 3.1.x
- Hadoop 3.2.0 and 3.2.1
- Hadoop 3.3.0
Incompatible releases have no problems reading data in stores
where markers are retained, but can get confused when deleting
or renaming directories.

If you are still using older versions to write to data, and cannot
yet upgrade, switch the option back to "delete"

Contributed by Steve Loughran
 2023-06-08 12:12:29 +01:00
hfutatzhanghb
0e6bd09ae3
HDFS-17003. Erasure Coding: invalidate wrong block after reporting bad blocks from datanode (#5643). Contributed by hfutatzhanghb. 
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-08 18:06:51 +08:00
hfutatzhanghb
ddae78b0ec
HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. (#5708). Contributed by farmmamba. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-08 14:28:01 +05:30
huhaiyang
0c209961f8
HDFS-17019. Optimize the logic for reconfigure slow peer enable for Namenode" (#5671) 
* HDFS-17019. Optimize the logic for reconfigure slow peer enable for Namenode
 2023-06-08 10:05:49 +08:00
Viraj Jasani
1dbaba8e70
HADOOP-18740. S3A prefetch cache blocks should be accessed by RW locks (#5675) 
Contributed by Viraj Jasani
 2023-06-07 14:05:52 +01:00
slfan1989
9de13f879a
YARN-11502. Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster. (#5705) 2023-06-05 15:54:41 -07:00
slfan1989
e6937d7076
YARN-11425. [Hotfix] YARN-11425. Modify Expiration Time Unit error. (#5712) 2023-06-05 15:51:39 -07:00
slfan1989
fd3c3ae068
YARN-11500. Fix typos in hadoop-yarn-server-common#federation. (#5702) 2023-06-05 15:49:36 -07:00
zhtttylz
d9980ab40f
HDFS-17029. Support getECPolices API in WebHDFS (#5698). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 17:33:37 +05:30
caozhiqiang
5d6ca13c5c
HDFS-16983. Fix concat operation doesn't honor dfs.permissions.enabled (#5561). Contributed by caozhiqiang. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 16:42:59 +05:30
slfan1989
241398de3b
YARN-11492. Improve createJerseyClient#setConnectTimeout Code. (#5636). Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 16:36:07 +05:30