big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,070 Commits 2 Branches 2 Tags 264 MiB
3f4221ec34
Commit Graph

2069 Commits

Author SHA1 Message Date
Shubham Gupta
3f4221ec34
MAPREDUCE-7351 - CleanupJob during handle of SIGTERM signal (#3176) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2021-07-07 09:08:15 +05:30
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Viraj Jasani
6e11461eaa
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-21 16:46:06 +09:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Viraj Jasani
207c92753f
MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project (#3074) 2021-06-07 11:51:29 +09:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Akira Ajisaka
8a489ce78e
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-05-26 15:47:56 +09:00
Viraj Jasani
2a206c20cc
HADOOP-17722. Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project (#3032) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:07:22 +09:00
Masatake Iwasaki
0d59500e8c HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 2021-05-21 01:36:26 +00:00
Ayush Saxena
5404ab4bca
MAPREDUCE-7343. Increase the job name max length in mapred job -list. (#2995). Contributed by Ayush Saxena. 2021-05-14 00:15:33 +05:30
Akira Ajisaka
1448756505
MAPREDUCE-7334. TestJobEndNotifier fails. (#2877) 2021-04-08 23:53:51 -07:00
lichaojacobs
663ca14a76
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x (#2775) 2021-04-09 11:58:53 +09:00
Sungpeo Kook
46a5979805
MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English. (#1942) 
Contributed by Sungpeo Kook.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-07 23:58:10 -07:00
Brahma Reddy Battula
03e42efa30 YARN-10544. AMParams.java having un-necessary access identifier static final. Contributed by ANANDA G B. 2021-03-31 08:25:20 +05:30
Jim Brennan
ede490d131 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 2021-03-22 18:41:25 +00:00
Jim Brennan
299b8062f1 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-15 20:13:17 +00:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Eric Badger
a17dc4e821 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.
 2021-03-02 21:47:31 +00:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Eric Badger
462561654b MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.
 2021-02-09 19:01:38 +00:00
Jungtaek Lim
2a38ed0e0c
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624) 
Contributed by Jungtaek Lim.
 2021-01-27 19:08:08 +00:00
lzx404243
ca7dd5fad3
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:45:13 +09:00
Steve Loughran
9b2956e254
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579) 
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null. 

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.
 2020-12-31 16:02:10 +00:00
adamantal
3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
Akira Ajisaka
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:32:10 +09:00
Akira Ajisaka
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488)" because JIRA ID is missing. 
This reverts commit ce5b3d7975.
 2020-11-26 11:30:53 +09:00
Akira Ajisaka
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:27:30 +09:00
Szilard Nemeth
f813f14b5b MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan 2020-11-24 17:02:35 +01:00
dengzh
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469) 
Contributed by Zhihua Deng.
 2020-11-23 15:40:22 +00:00
dengzh
740399a602
HADOOP-17390. Skip license check on lz4 code files (#2478) 
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
 2020-11-20 14:12:20 +00:00
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Liang-Chi Hsieh
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-11-18 12:03:25 -08:00
Akira Ajisaka
2045a9df13
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. (#2463) 2020-11-18 14:15:11 +09:00
Peter Bacsko
da93cd962c MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 2020-10-28 10:35:30 +01:00
Akira Ajisaka
7bc305db5d
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 2020-10-23 04:34:49 +09:00
hemanthboyina
42e7e0380f HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Swaroopa Kadam
2e46ef9417
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 10:57:02 -07:00
Liang-Chi Hsieh
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 
This switches the SnappyCodec to use the java-snappy codec, rather than the native one. 

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh
 2020-10-06 17:07:54 +01:00
Ikko Ashimine
4347a5c955
HADOOP-17294. Fix typos existance to existence (#2357) 2020-10-06 10:10:44 +09:00
zz
95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
Szilard Nemeth
4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Adam Antal
82a7505646 YARN-10304. Create an endpoint for remote application log directory path query. Contributed by Andras Gyori 2020-08-25 13:29:12 +02:00
ywheel
cf4eb75608
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 2020-07-30 13:01:22 +09:00
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Masatake Iwasaki
80046d1c8a
MAPREDUCE-7285. Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar. (#2139) 2020-07-15 15:24:34 +09:00
Akira Ajisaka
3e7b65c3e5
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 2020-07-15 13:46:20 +09:00