big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,224 Commits 2 Branches 2 Tags
Commit Graph

5911 Commits

Author SHA1 Message Date
Steve Loughran
a505940a2f
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:41 +01:00
Viraj Jasani
20d3b9cc46
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) (#5374) 2023-03-30 07:13:10 +08:00
slfan1989
4a23389f5e
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-03-10 15:16:28 +00:00
Varun Saxena
157af0cb22 YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary

(cherry picked from commit 2a0dc2ab2f5fb46dc540ed440d6c8b2896dd195b)
 2023-03-05 23:02:17 +05:30
Steve Loughran
b9b999f7b1
YARN-11441. Revert YARN-10495. 
This reverts commit 1022ed91ac4242d05246e03275572bd4d8b225ed.
 2023-02-17 15:07:41 +00:00
Chris Nauroth
1f270d8a5e YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
(cherry picked from commit 6b67373d10717602da18ae701e8bc7f20dbb54d6)
 2022-12-28 19:02:02 +00:00
curie71
290dc7817c YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 9668a85d40a6a98514a24d5f25ab757501fe3423)
 2022-12-28 00:00:18 +00:00
Bence Kosztolnik
7190fcf713 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:20:35 +01:00
susheel-gupta
f9fac84f43 YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:26:05 +01:00
K0K0V0K
8b748c1cb8 YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit ee7d1787cd2cdfea936425dcf88e750827ff5c1b)
 2022-12-08 18:15:08 +00:00
Melissa You
853ffb545a
HADOOP-18515. Backport HADOOP-17612 to branch-3.3(Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0) (#5097) 
* HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
 2022-11-05 09:28:24 -07:00
Ashutosh Gupta
7b84f6458b
HADOOP-18484. Upgrade hsqldb to v2.7.1 to mitigate CVE-2022-41853 (#5101) 2022-11-04 11:00:17 +01:00
Ashutosh Gupta
0961014262 YARN-11364. Docker Container to accept docker Image name with sha256 digest (#5092) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 83acb559817a97c14c4e3fd846dcc16ab615093e)
 2022-11-01 21:45:04 +00:00
Ashutosh Gupta
c5830d237b YARN-11363. Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests (#5091) 
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 69225ae5b9b4153e1c2ba281fd0a09cc1514a962)
 2022-11-01 21:10:46 +00:00
Chris Nauroth
33293d4ba4 YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 
(cherry picked from commit bfb84cd7f66fb6fe98809cc5d6c59864995855b1)
 2022-10-28 18:13:57 +00:00
Steve Loughran
19f8e4f34d
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-21 14:16:22 +01:00
PJ Fanning
ea851c5e4a
HADOOP-15983. Use jersey-json that is built to use jackson2 ((#3988) 
Moves from com.sun.jersey 1.19 to the artifact
com.github.pjfanning:jersey-json:1.20

This allows jackson 1 to be removed from the classpath.

Contains

* HADOOP-16908. Prune Jackson 1 from the codebase and restrict
   its usage for future
* HADOOP-18219. Fix shaded client test failure

These are needed for the HADOOP-15983 changes to build.

Contributed by PJ Fanning.
 2022-10-20 17:37:56 +01:00
slfan1989
2e3f91bdf5
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-17 10:23:13 +05:30
Ashutosh Gupta
1c3bf42ad0
YARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895) 
Contributed by Ashutosh Gupta
 2022-10-05 12:09:11 +01:00
Ashutosh Gupta
2532eca013
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 65a027b11299ac2d57556406a614442d8fc9acd4)
 2022-09-12 23:33:10 +09:00
Ashutosh Gupta
3b3bd89084 YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING (#4721) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f02ff1afe2eed4edab4fae6c562102fdca9a6893)
 2022-08-16 19:07:42 +09:00
Steve Loughran
9c5228cf6b
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.

Change-Id: I98b0fed54da3b810c3f23fe5b12e673937916257
 2022-08-05 14:02:28 +01:00
Masatake Iwasaki
ff13f9ee8b Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57dfd41385dbce5de90620c48a45abb263)
 2022-07-22 02:31:34 +00:00
PJ Fanning
6733ba56b8
HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7. (#4552) 
This downgrades jackson from the version switched to in 
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.

The 2.12.7 release still contains the fix for CVE-2020-36518.

Contributed by PJ Fanning
 2022-07-16 18:18:52 +01:00
Szilard Nemeth
ebe3ed5cd3 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 
(cherry picked from commit 7cb887e6c2bfc058bb7b7d7bde2554d261432bdb)
 2022-07-07 21:04:15 +00:00
Benjamin Teke
838b63d836 YARN-10974. Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu. 2022-06-22 18:20:09 +02:00
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482) 
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
 2022-06-22 13:09:50 +01:00
zhengchenyu
d7de378b22
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. (#4408) 
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.

Contributed by zhengchenyu
 2022-06-17 19:51:56 +01:00
Steve Loughran
de9f994338
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi (#4417) 
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.

Contributed by Steve Loughran

Change-Id: I61d087041561eeb8c9c42b5b7d8f0bb63f296b15
 2022-06-10 17:03:25 +01:00
Ashutosh Gupta
e0732baeb8 YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e3e9369c1d7c3045f910a7c320970b643a73c862)
 2022-05-26 15:58:47 +09:00
Owen O'Malley
1f111d6a41 YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 
(cherry picked from commit f390edaec44cfa91b2b09549091f033f1749d8ac)
 2022-05-24 05:17:34 +00:00
Szilard Nemeth
90ec4418c7 YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-18 14:34:08 +02:00
Szilard Nemeth
4f112e3138 YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-18 14:25:35 +02:00
Szilard Nemeth
b4550b3356 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-18 14:08:41 +02:00
Zilong Zhu
2dd5852ad0 YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280) 
(cherry picked from commit 79cc0caf9fde8ad6a9170ed7572a73e01b37d9e0)
 2022-05-18 03:03:13 +09:00
Steve Loughran
fe306ce57e HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:09 +01:00
Ashutosh Gupta
a50ce18f4d YARN-11092. Upgrade jquery ui to 1.13.1 (#4260) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 931abbd92706aa5377ba303b7b09f0adf1d8e1c7)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java
 2022-05-16 23:26:46 +09:00
Ashutosh Gupta
d58f9d438f
YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6985f9aabebefa86037f7dbc478374392afef990)
 2022-05-14 01:41:31 +09:00
Ashutosh Gupta
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5beac5e1f2da5907162d9438e98b98af)
 2022-05-08 07:39:17 +09:00
jteagles
f155abc572
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d4a91bd0c0d9d438ffe95fafee0212499b0abcfd)
 2022-05-03 03:15:10 +09:00
Ashutosh Gupta
4f9e607126
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a74acc755edcde38203416f9d5225343ea877c08)
 2022-05-03 03:10:43 +09:00
Steve Loughran
e123de9f19
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
 2022-04-27 19:23:25 +01:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
Steve Loughran
caecec45f5
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939) 
Reviewed-by: Siyao Meng <siyao@apache.org>

Contributed by Viraj Jasani
 2022-04-20 16:36:51 +01:00
Akira Ajisaka
2112ef61e0
YARN-10553. Refactor TestDistributedShell (#4159) 
(cherry picked from commit 890f2da624465473a5f401a3bcfc4bbd068289a1)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDSWithMultipleNodeManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java

Co-authored-by: Ahmed Hussein <50450311+amahussein@users.noreply.github.com>
 2022-04-13 12:22:17 +09:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00
Peter Bacsko
ab5ea272ec YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 
(cherry picked from commit a0deda1a777d8967fb8c08ac976543cda895773d)
 2022-03-24 14:11:23 +09:00
Siyao Meng
4153c16324 YARN-10747. Bump YARN CSI protobuf version to 3.7.1 (#2946) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ib2a4bf775a5804641a91c6f9a486772c9d87c17e
(cherry picked from commit 5b09ba75eccab4c2f1850bcc4cd8e60241cb500e)
 2022-03-22 14:46:45 +09:00
Fei Hui
5a38ed2f22 HADOOP-17276. Extend CallerContext to make it include many items (#2327) 
Cherry-picked from d0d10f7e by Owen O'Malley
 2022-03-14 10:28:38 -07:00