big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,668 Commits 2 Branches 2 Tags 264 MiB
03a499821c
Commit Graph

26668 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
Steve Loughran
160b9fc3c9
HADOOP-18755. openFile builder new optLong() methods break hbase-filesystem (#5704) 
This is a followup to 
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem

Contributed by Steve Loughran
 2023-06-01 14:31:08 +01:00
smarthan
9f1e23cc67
HDFS-17031. Reduce some repeated codes in RouterRpcServer. (#5701). Contributed by Chengwei Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-01 08:32:38 +05:30
NishthaShah
f8b7ddf69c
HDFS-16996. Fix flaky testFsCloseAfterClusterShutdown in TestFileCreation (#5697). Contributed by Nishtha Shah. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-01 08:23:48 +05:30
Simbarashe Dzinamarira
d92a5815f4
HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) 2023-05-31 10:20:19 -07:00
Marcono1234
9acf462d26
HDFS-17000. Fix faulty loop condition in TestDFSStripedOutputStreamUpdatePipeline (#5699). Contributed by Marcono1234. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-31 16:10:07 +05:30
hchaverri
124313d215
HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency. (#5691). Contributed by hchaverri. 
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-31 14:26:31 +08:00
slfan1989
86c250a54a
YARN-7720. Race condition between second app attempt and UAM timeout when first attempt node is down. (#5672) 2023-05-29 10:37:08 -07:00
Xianming Lei
97afb33c73
YARN-11276. Add LRU cache for RMWebServices.getApps. (#4793) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-05-26 20:46:00 +08:00
slfan1989
b977065cc4
YARN-11478. [Federation] SQLFederationStateStore Support Store ApplicationSubmitData. (#5663) 2023-05-24 11:43:20 -07:00
Steve Loughran
e6b54f7f68
Revert "HADOOP-18706. Improve S3ABlockOutputStream recovery (#5563)" 
This reverts commit 372631c566.

Reverted due to HADOOP-18744.
 2023-05-24 19:22:22 +01:00
hfutatzhanghb
e9740cb17a
HDFS-16908. Fix javadoc of field IncrementalBlockReportManager#readyToSend. (#5351). Contributed by farmmamba. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-23 16:12:50 +05:30
Tamas Domok
aeb3f6f1a8
YARN-11490. Reverting YARN-11211 and eliminating the use of DefaultMetricsSystem during configuration validation (#5644) 2023-05-23 10:36:37 +02:00
Ashutosh Gupta
a98d15804a
MAPREDUCE-7419. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-common (#5028). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-23 11:37:49 +05:30
Viraj Jasani
3b65b5d68f
HDFS-17020. RBF: mount table addAll should print failed records in std error (#5674) 2023-05-22 18:39:52 -07:00
Gautham B A
afe850ca2c
HADOOP-18746. Install Python 3 for Windows 10 docker image (#5679) 
* This PR installs Python 3.10.11 for
  Windows 10 Docker image to fix
  the issue with building mvnsite.
* After installing Python 3.10.11, it
  creates the hardlink python -> python3
  as required by the script.
 2023-05-21 21:10:04 +05:30
hfutatzhanghb
5b22dc6ace
HDFS-16909. Improve ReplicaMap#mergeAll method. (#5353). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-21 19:08:58 +08:00
huhaiyang
af933f3a4f
HDFS-17017. Fix the issue of arguments number limit in report command in DFSAdmin (#5667). Contributed by Haiyang Hu. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-21 09:03:34 +05:30
Xianming Lei
0110e24ed8
YARN-11496. Improve TimelineService log format. (#5677). Contributed by Xianming Lei. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 14:57:45 +05:30
NishthaShah
9a524ede87
HDFS-17022. Fix the exception message to print the Identifier pattern (#5678). Contributed by Nishtha Shah. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 14:40:23 +05:30
zhtttylz
408dbf318e
HDFS-17014. HttpFS Add Support getStatus API (#5660). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 14:37:38 +05:30
NishthaShah
5272ed8670
HADOOP-17518. Update the regex to A-Z (#5669). Contributed by Nishtha Shah. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 06:21:13 +05:30
Keyao Li
0914b3e792
HDFS-16697. Add logs if resources are not available in NameNodeResourcePolicy. (#5569). Contributed by ECFuzz. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 18:38:01 +05:30
Keyao Li
339bc7b3a6
HDFS-16653. Improve error messages in ShortCircuitCache. (#5568). Contributed by ECFuzz. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 07:43:18 +05:30
Xianming Lei
441fb23293
HDFS-17018. Improve dfsclient log format. (#5668). Contributed by Xianming Lei. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 06:25:43 +05:30
Patrick GRANDJEAN
4627242c44
HADOOP-18652. Path.suffix raises NullPointerException (#5653). Contributed by Patrick Grandjean. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:16:55 +05:30
LiuGuH
f6770dee47
HDFS-16979. RBF: Add proxyuser port in hdfsauditlog (#5552). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:02:16 +05:30
slfan1989
bba663038d
YARN-8898. Fix FederationInterceptor#allocate to set application priority in allocateResponse. (#5645) 2023-05-18 11:57:38 -07:00
Peter Szucs
ff8eac517a
YARN-11463. Node Labels root directory creation doesn't have a retry logic - 2nd addendum (#5670) 2023-05-18 14:48:43 +02:00
jianghuazhu
78cc528739
HDFS-17012.Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT. (#5659). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 14:27:19 +05:30
Tsz-Wo Nicholas Sze
350dcaf616
HDFS-17010. Add a subtree test to TestSnapshotDiffReport. (#5656) 2023-05-18 15:53:26 +08:00
wangzhaohui
03163f9de2
HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 12:12:11 +05:30
slfan1989
5ddaf2e133
YARN-11493. [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router. (#5651) 2023-05-17 11:09:10 -07:00
Viraj Jasani
8e17385141
HDFS-17009. RBF: state store putAll should also return failed records (#5664) 2023-05-17 09:33:34 -07:00
liang3zy22
482897a0f6
Fix typos in HDFS documents. (#5665) 2023-05-17 09:28:01 -07:00
Steve Loughran
a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport (#5662) 
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()

This brings trunk into sync with the branch-3.3 changes
 2023-05-16 18:16:24 +01:00
Viraj Jasani
bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops (#5621) 
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter

Contributed by Viraj Jasani
 2023-05-16 10:40:16 +01:00
Chun Chen
11af08d67a
YARN-11489. Fix memory leak of DelegationTokenRenewer futures in DelegationTokenRenewerPoolTracker. (#5629). Contributed by Chun Chen. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:38:04 +08:00
smarthan
251439d769
HDFS-16985. Fix data missing issue when delete local block file. (#5564). Contributed by Chengwei Wang. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:33:38 +08:00
slfan1989
e0938b4c2a
YARN-11495. Fix typos in hadoop-yarn-server-web-proxy. (#5652). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 11:41:38 +05:30
slfan1989
2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). (#5625) 2023-05-12 18:12:05 -07:00
Viraj Jasani
5d0cc455f5
HDFS-17008. Fix RBF JDK 11 javadoc warnings (#5648) 2023-05-12 17:27:13 -07:00
zhtttylz
0c77629849
HDFS-17001. Support getStatus API in WebHDFS (#5628). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 03:49:13 +05:30
Steve Loughran
ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods (#5650) 
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
 2023-05-12 15:22:37 +01:00
susheel-gupta
0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade (#5649) 2023-05-12 14:20:26 +02:00
WangYuanben
905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. (#5520). Contributed by WangYuanben. 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-12 04:12:02 +05:30
Steve Loughran
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
Viraj Jasani
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points (#5554). Contributed by Viraj Jasani. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-11 08:45:34 +05:30
zhtttylz
5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API (#5602). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 23:10:19 +05:30
slfan1989
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00