big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,196 Commits 2 Branches 2 Tags 264 MiB
38b2ed2151
Commit Graph

26196 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
monthonk
1f157f802d
HADOOP-17386. Change default fs.s3a.buffer.dir to be under Yarn container path on yarn applications (#3908) 
Co-authored-by: Monthon Klongklaew <monthonk@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-22 13:50:27 +09:00
Viraj Jasani
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
Viraj Jasani
697e5d4636
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-02-22 11:00:38 +08:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Neil
2f448acf39
HDFS-16459. RBF: register RBFMetrics in MetricsSystem for promethuessink (#3999) 2022-02-21 11:32:34 -08:00
Steve Loughran
cae749b076
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I73af19d2e2e41f4ba686c470726a80c3903a1950
 2022-02-21 17:08:56 +00:00
Wei-Chiu Chuang
007c2011ef
YARN-11068. Update transitive log4j2 dependency to 2.17.1 (#3963) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-21 13:33:38 +09:00
Mohanad Elsafty
a4f459097b
HADOOP-18117. Add an option to preserve root directory permissions (#3970) 2022-02-18 19:12:50 +08:00
Szilard Nemeth
de526e166c YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-02-17 21:05:01 +01:00
Szilard Nemeth
7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth
51001fc7bf YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok 2022-02-17 19:38:18 +01:00
Szilard Nemeth
48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
PJ Fanning
5f6a294fab
HADOOP-18126. update junit 5 version due to build issues (#3993) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-17 14:03:39 +09:00
jianghuazhu
2025243fbf
HADOOP-18082.Add debug log when RPC#Reader gets a Call. (#3891). Contributed by JiangHua Zhu. 2022-02-17 01:49:45 +05:30
Chentao Yu
19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
YulongZ
48bef285a8
HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971) 2022-02-15 08:44:17 -08:00
litao
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 13:40:46 +09:00
Akira Ajisaka
7fd90cdcbe
YARN-10788. TestCsiClient fails (#3989) 
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 01:13:13 +09:00
GuoPhilipse
b68964336d
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-14 05:38:28 +09:00
daimin
0e74f1e467
Fix thread safety of EC decoding during concurrent preads (#3881) 2022-02-11 10:20:00 +08:00
Xing Lin
ca8ba24051 HADOOP-18110. ViewFileSystem: Add Support for Localized Trash Root 
Fixes #3956
 2022-02-10 16:43:04 -08:00
Ayush Saxena
fe583c4b63
HADOOP-18096. Distcp: Sync moves filtered file to home directory rather than deleting. (#3940). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: stack <stack@apache.org>
 2022-02-11 01:59:40 +05:30
Neil
c777142ed3
RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. (#3965) 2022-02-10 09:17:30 -08:00
Steve Loughran
efdec92cab
HADOOP-18091. S3A auditing leaks memory through ThreadLocal references (#3930) 
Adds a new map type WeakReferenceMap, which stores weak
references to values, and a WeakReferenceThreadMap subclass
to more closely resemble a thread local type, as it is a
map of threadId to value.

Construct it with a factory method and optional callback
for notification on loss and regeneration.

 WeakReferenceThreadMap<WrappingAuditSpan> activeSpan =
      new WeakReferenceThreadMap<>(
          (k) -> getUnbondedSpan(),
          this::noteSpanReferenceLost);

This is used in ActiveAuditManagerS3A for span tracking.

Relates to
* HADOOP-17511. Add an Audit plugin point for S3A
* HADOOP-18094. Disable S3A auditing by default.

Contributed by Steve Loughran.
 2022-02-10 12:31:41 +00:00
Neil
390967f1f0
RBF: Add search box in Router's tab-mounttable web page (#3968) 2022-02-09 09:17:23 -08:00
Joey Krabacher
a08e69d33e
HADOOP-18114. Documentation correction in assumed_roles.md (#3949) 
Fixes typo in hadoop-aws/assumed_roles.md

Contributed by Joey Krabacher
 2022-02-09 10:35:11 +00:00
secfree
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847) 2022-02-09 10:07:05 +08:00
YulongZ
3e7a7c3c4a
HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878) 2022-02-08 08:40:57 -08:00
Viraj Jasani
b39b334873
HADOOP-18098. Basic verification for the release candidate vote (#3944) 2022-02-07 09:34:58 +08:00
singer-bin
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 2022-02-06 13:05:57 +08:00
Petre Bogdan Stolojan
5e7ce26e66
HADOOP-18085. S3 SDK Upgrade causes AccessPoint ARN endpoint mistranslation (#3902) 
Part of HADOOP-17198. Support S3 Access Points.

HADOOP-18068. "upgrade AWS SDK to 1.12.132" broke the access point endpoint
translation.

Correct endpoints should start with "s3-accesspoint.", after SDK upgrade they start with
"s3.accesspoint-" which messes up tests + region detection by the SDK.

Contributed by Bogdan Stolojan
 2022-02-04 15:37:08 +00:00
Abhishek Das
3684c7f66a
HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>
 2022-02-03 16:28:04 -08:00
Aswin Shakil Balasubramanian
41c86b6464
HADOOP-18101. Bump aliyun-sdk-oss to 3.13.2 and jdom2 to 2.0.6.1 (#3951) 2022-02-03 15:50:38 -08:00
litao
b5b07af605
HDFS-16435. Remove no need TODO comment for ObserverReadProxyProvider (#3912). Contributed by tomscut. 
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-03 08:21:06 +05:30
Gautham B A
ed44662968
HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945) 2022-02-02 11:12:17 +05:30
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
Ayush Saxena
aeae5716cc
Revert "HADOOP-18024. SocketChannel is not closed when IOException happens in Server$Listener.doAccept (#3719)" 
This reverts commit 6ed01585eb.

Breaks TestIPC#testIOEOnListenerAccept
 2022-02-01 14:11:25 +05:30
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741) 
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-02-01 12:18:54 +05:30
KevinWikant
089e06de21
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:04:32 +09:00
litao
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web (#3943) 2022-01-30 16:13:58 +08:00
secfree
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 17:04:11 +09:00
Akira Ajisaka
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2022-01-28 15:51:21 +09:00
Akira Ajisaka
02f6bad1ff Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936)" 
This reverts commit 1c01944f35.
 2022-01-28 00:36:25 +09:00
Wei-Chiu Chuang
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 00:04:01 +09:00
Li MingXiang
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00