big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,278 Commits 2 Branches 2 Tags 264 MiB
b43901c271
Commit Graph

1374 Commits

Author SHA1 Message Date
zeekling
b43901c271
添加Capacity调度器的源码注释 2024-11-08 23:21:45 +08:00
Wei-Chiu Chuang
8af2d2feb2
Update version to 3.3.6 2023-06-12 15:34:41 -07:00
Dongjoon Hyun
20d073cb2c
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. 
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit fb16e00da0)

Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2023-06-12 10:40:41 -07:00
Steve Loughran
ab594ec77e
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-16 13:41:17 +01:00
Bence Kosztolnik
7190fcf713 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:20:35 +01:00
Chris Nauroth
33293d4ba4 YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 
(cherry picked from commit bfb84cd7f6)
 2022-10-28 18:13:57 +00:00
PJ Fanning
ea851c5e4a
HADOOP-15983. Use jersey-json that is built to use jackson2 ((#3988) 
Moves from com.sun.jersey 1.19 to the artifact
com.github.pjfanning:jersey-json:1.20

This allows jackson 1 to be removed from the classpath.

Contains

* HADOOP-16908. Prune Jackson 1 from the codebase and restrict
   its usage for future
* HADOOP-18219. Fix shaded client test failure

These are needed for the HADOOP-15983 changes to build.

Contributed by PJ Fanning.
 2022-10-20 17:37:56 +01:00
Ashutosh Gupta
1c3bf42ad0
YARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895) 
Contributed by Ashutosh Gupta
 2022-10-05 12:09:11 +01:00
Ashutosh Gupta
2532eca013
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 65a027b112)
 2022-09-12 23:33:10 +09:00
PJ Fanning
6733ba56b8
HADOOP-18332. Remove rs-api dependency by downgrading jackson to 2.12.7. (#4552) 
This downgrades jackson from the version switched to in 
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.

The 2.12.7 release still contains the fix for CVE-2020-36518.

Contributed by PJ Fanning
 2022-07-16 18:18:52 +01:00
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482) 
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
 2022-06-22 13:09:50 +01:00
Szilard Nemeth
b4550b3356 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-18 14:08:41 +02:00
Zilong Zhu
2dd5852ad0 YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280) 
(cherry picked from commit 79cc0caf9f)
 2022-05-18 03:03:13 +09:00
Ashutosh Gupta
a50ce18f4d YARN-11092. Upgrade jquery ui to 1.13.1 (#4260) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 931abbd927)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java
 2022-05-16 23:26:46 +09:00
jteagles
f155abc572
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d4a91bd0c0)
 2022-05-03 03:15:10 +09:00
Steve Loughran
e123de9f19
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
 2022-04-27 19:23:25 +01:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00
Peter Bacsko
ab5ea272ec YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 
(cherry picked from commit a0deda1a77)
 2022-03-24 14:11:23 +09:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
Ashutosh Gupta
1c99810b89 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)
 2021-12-23 17:16:09 +09:00
Viraj Jasani
b0c1158829
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3764) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-13 13:52:44 +09:00
Prabhu Joseph
edfd68e248 YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09c)
 2021-12-06 14:45:18 +09:00
Chao Sun
e079fa6577 Preparing for 3.3.3 development 2021-11-16 16:02:34 -08:00
Viraj Jasani
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6)
 2021-10-07 11:00:19 +09:00
Peter Bacsko
99d84b941b YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient. Contributed by Bilwa S T. 2021-09-23 13:58:56 +02:00
Tamas Domok
8e4ac01135
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3409) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:34:32 +02:00
9uapaw
f001595b47 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a)
 2021-07-28 17:12:40 +02:00
Jim Brennan
57ee4238eb YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit 9a6a11c452)
 2021-06-15 19:30:16 +00:00
Jim Brennan
48aa23eb3b YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 
(cherry picked from commit c4581827a9)
 2021-05-25 18:10:24 +00:00
Szilard Nemeth
6777acb08c YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:36:51 +02:00
Wei-Chiu Chuang
fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
Aryan Gupta
5f147112cf
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 
(cherry picked from commit bbbfa7d415)
 2021-05-13 12:16:20 +08:00
Aryan Gupta
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
 2021-05-13 12:16:17 +08:00
Peter Bacsko
051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Eric Badger
da2ebfa8a9 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 
(cherry picked from commit 803ac4b1a0)
 2021-04-29 17:52:03 +00:00
Eric Badger
003deeeecf YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 22:14:57 +00:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 23:34:35 +00:00
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
Eric Payne
f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:14:19 +00:00
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:20:37 +00:00
Peter Bacsko
eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Eric Badger
e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:07:59 +00:00
Eric Badger
305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 18:59:45 +00:00
Eric Badger
27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:25:25 +00:00
Prabhu Joseph
72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0b)
 2021-02-18 14:25:16 +05:30
Prabhu Joseph
0c46ab51b5 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850)
 2021-02-18 13:59:50 +05:30
Jim Brennan
d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af43)
 2021-02-05 17:37:04 +00:00
bibinchundatt
1520b84b36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 
(cherry picked from commit 8bc2dfbf36)
 2021-01-22 08:30:12 +05:30