big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,951 Commits 2 Branches 2 Tags 264 MiB
5eeab5e1b9
Commit Graph

776 Commits

Author SHA1 Message Date
Benjamin Teke
85b3ea6f50
MAPREDUCE-7449: Add add-opens flag to container launch commands on JDK17 nodes (#5935) 2023-08-10 22:47:17 +02:00
yu liang
caf1816e0e
MAPREDUCE-7442. Exception message is not intusive when accessing the job configuration web UI (#5848) 2023-07-24 21:40:36 +08:00
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c.
 2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
Ayush Saxena
fe5bb49ad9
Revert "YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta" 
This reverts commit 8eda456d37.
 2023-02-22 07:28:13 +05:30
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Szilard Nemeth
815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth
8eda456d37 YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta 2023-01-25 15:06:20 +01:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Ashutosh Gupta
85ec7969a7
MAPREDUCE-7428. Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp (#5209) 
Contributed by: Ashutosh Gupta
 2022-12-14 12:54:08 +00:00
Samrat
e04c9e810b
MAPREDUCE-7426. Fix typo in StartEndTimeBase (#4894) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:23:30 +09:00
PJ Fanning
aac87ffe76
MAPREDUCE-7411: use secure XML parsers in mapreduce modules (#4980) 
Lockdown of parsers in hadoop-mapreduce.

Follow-on to HADOOP-18469. Add secure XML parser factories to XMLUtils

Contributed by P J Fanning
 2022-10-21 14:02:11 +01:00
Ashutosh Gupta
59d3c20118
MAPREDUCE-7407. Avoid stopContainer() on dead node (#4779) 2022-09-15 10:30:36 -07:00
Ashutosh Gupta
4e8c0b902e
MAPREDUCE-7201.Make Job History File Permissions configurable (#4507) 
* MAPREDUCE-7201.Make Job History File Permissions configurable

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-11 11:34:52 +05:30
Ashutosh Gupta
36c4be819f
MAPREDUCE-7369. Fixed MapReduce tasks timing out when spends more time on MultipleOutputs#close (#4247) 
Contributed by Ravuri Sushma sree.

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-20 17:01:01 +09:00
Ashutosh Gupta
9c3330c22f
MAPREDUCE-7377. Remove unused imports in MapReduce project (#4299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:34:19 +09:00
Szilard Nemeth
f143e99428 MAPREDUCE-7379. RMContainerRequestor#makeRemoteRequest has confusing log message. Contributed by Ashutosh Gupta 2022-05-11 16:55:19 +02:00
Steve Loughran
6999acf520
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
 2022-04-24 17:33:05 +01:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
Viraj Jasani
b1ad4eab9a
HADOOP-17959. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules (#3537) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 16:22:50 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
Eric Payne
7581413156 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 2021-07-07 20:43:44 +00:00
Shubham Gupta
3f4221ec34
MAPREDUCE-7351 - CleanupJob during handle of SIGTERM signal (#3176) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2021-07-07 09:08:15 +05:30
Viraj Jasani
6e11461eaa
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-21 16:46:06 +09:00
Viraj Jasani
207c92753f
MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project (#3074) 2021-06-07 11:51:29 +09:00
Viraj Jasani
2a206c20cc
HADOOP-17722. Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project (#3032) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:07:22 +09:00
Brahma Reddy Battula
03e42efa30 YARN-10544. AMParams.java having un-necessary access identifier static final. Contributed by ANANDA G B. 2021-03-31 08:25:20 +05:30
Jim Brennan
299b8062f1 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-15 20:13:17 +00:00
lzx404243
ca7dd5fad3
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:45:13 +09:00
Szilard Nemeth
f813f14b5b MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan 2020-11-24 17:02:35 +01:00
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Ikko Ashimine
4347a5c955
HADOOP-17294. Fix typos existance to existence (#2357) 2020-10-06 10:10:44 +09:00
zz
95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Wilfred Spiegelenburg
10db97df1c
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.
 2020-05-28 13:23:01 +10:00
Surendra Singh Lilhore
d4e36409d4 MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T. 2020-05-19 11:06:36 +05:30
Eric E Payne
11d17417ce MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 2020-04-13 18:20:07 +00:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Ahmed Hussein
08251538fe MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-28 10:57:33 -06:00
Ahmed Hussein
3f01c48106 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-27 15:50:13 -06:00
Masatake Iwasaki
a2fdd7c2b5 MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804) 2020-01-15 12:09:09 +09:00
Ahmed Hussein
cdd6efd3ab MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-08 11:08:13 -06:00
Sunil G
682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
prabhujoseph
013c6e51b6 MAPREDUCE-7249. Fix Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure. 
Contributed by Wilfred Spiegelenburg.
 2019-11-28 10:57:15 +05:30
prabhujoseph
7f2ea2ac46 MAPREDUCE-7240. Fix Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER. 
Contributed by Huachao and Peter Bacsko. Reviewed by Wilfred Spiegelenburg.
 2019-11-27 14:27:27 +05:30
Ahmed Hussein
ed302f1fed MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 14:55:20 -06:00
Tao Yang
5d497abe21 YARN-9817. Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero. Contributed by Prabhu Joseph. 2019-09-07 07:39:02 +08:00
Szilard Nemeth
ac6c4f0b29 MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal 2019-08-12 13:54:28 +02:00