big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,671 Commits 2 Branches 2 Tags 264 MiB
1d0c9ab433
Commit Graph

943 Commits

Author SHA1 Message Date
Jungtaek Lim
2a38ed0e0c
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624) 
Contributed by Jungtaek Lim.
 2021-01-27 19:08:08 +00:00
Steve Loughran
9b2956e254
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579) 
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null. 

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.
 2020-12-31 16:02:10 +00:00
dengzh
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469) 
Contributed by Zhihua Deng.
 2020-11-23 15:40:22 +00:00
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Peter Bacsko
da93cd962c MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 2020-10-28 10:35:30 +01:00
Akira Ajisaka
7bc305db5d
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 2020-10-23 04:34:49 +09:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
zz
95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
ywheel
cf4eb75608
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 2020-07-30 13:01:22 +09:00
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Eric Badger
fbb8775430 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit e2322e1117.
 2020-06-08 20:35:27 +00:00
Eric E Payne
e2322e1117 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 2020-04-27 19:10:00 +00:00
Surendra Singh Lilhore
a1b0697d37 MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T 2020-04-18 19:42:20 +05:30
Eric E Payne
11d17417ce MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 2020-04-13 18:20:07 +00:00
Jason Lowe
c613296dc8 MAPREDUCE-7241. FileInputFormat listStatus with less memory footprint. Contributed by Zhihua Deng 2020-04-01 07:46:33 -05:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Wanqiang Ji
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters (#1397) 2020-03-16 11:28:36 +09:00
Sergey Pogorelov
b343e1533b MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793) 2020-01-06 12:36:11 +09:00
Ahmed Hussein
ed302f1fed MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 14:55:20 -06:00
Steve Loughran
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3 
Contributed by Steve Loughran.

Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
 2019-10-01 18:11:05 +01:00
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Szilard Nemeth
ac6c4f0b29 MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal 2019-08-12 13:54:28 +02:00
Szilard Nemeth
a7371a779c MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 13:01:30 +02:00
Mehul Garnara
c0a0c353e8
MAPREDUCE-6973. Fix comments on creating _SUCCESS file. 
This closes #280

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-26 21:21:26 +09:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Wanqiang Ji
e7e30a5f8b
MAPREDUCE-7210. Replace mapreduce.job.counters.limit with mapreduce.job.counters.max in mapred-default.xml 
This closes #878

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-05 10:45:23 +09:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Akira Ajisaka
5565f2c532
MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work. 2019-05-23 10:21:11 +09:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
David Mollitor
246ab77f28
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>
 2019-03-22 10:26:24 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Eric Yang
1ab69a9543 YARN-9221. Added flag to disable dynamic auxiliary service feature. 
Contributed by Billie Rinaldi
 2019-01-25 19:05:36 -05:00
Eric Yang
2fa9389c2e YARN-9146. Added REST API to configure auxiliary service. 
Contributed by Billie Rinaldi
 2019-01-22 18:24:43 -05:00
Steve Loughran
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran and Daryn Sharp.
 2019-01-14 17:59:27 +00:00
Akira Ajisaka
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens." 
This reverts commit d7152332b3.
 2019-01-08 14:51:30 +09:00
Steve Loughran
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran.
 2019-01-07 13:18:03 +00:00
Eric Yang
7affa3053c YARN-9131. Updated document usage for dynamic auxiliary service. 
Contributed by Billie Rinaldi
 2018-12-20 18:45:49 -05:00
Akira Ajisaka
e0d75088f2
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin. 2018-12-13 16:26:25 +09:00
Jason Lowe
4d8de7ab69 MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla 2018-11-28 14:54:59 -06:00
Akira Ajisaka
13a21f6607
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin. 2018-11-28 17:57:42 +09:00
Akira Ajisaka
1aad99a718
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin. 2018-11-27 20:01:11 +09:00
Wangda Tan
762a56cc64 MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda) 
Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a
 2018-11-13 11:25:41 -08:00
Jason Lowe
0b6625a973 MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan 2018-11-07 08:20:49 -06:00
Haibo Chen
823bb5dda8 MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter) 2018-10-23 15:28:13 -07:00
Haibo Chen
babd1449bf MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev) 2018-10-16 13:44:41 -07:00
Xiao Chen
25f8fcb064 MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko. 2018-10-16 10:23:31 -07:00
Steve Loughran
735a36afd5
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection. 
Contributed by Steve Loughran.
 2018-10-11 20:01:42 +01:00
Akira Ajisaka
5b1cfcaeff
MAPREDUCE-7035. Skip javadoc build for auto-generated sources in hadoop-mapreduce-client. Contributed by Mukul Kumar Singh. 2018-10-09 21:13:04 +09:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Wangda Tan
230d90c5b9 MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda) 
Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a
 2018-09-24 13:52:53 -07:00
Jason Lowe
2886024ac3 MAPREDUCE-7133. History Server task attempts REST API returns invalid data. Contributed by Oleksandr Shevchenko 2018-09-13 14:41:38 -05:00
Steve Loughran
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.
 2018-08-30 14:49:53 +01:00
Haibo Chen
84973d1049 MAPREDUCE-6861. Add metrics tags for ShuffleClientMetrics. (Contributed by Zoltan Siegl) 2018-08-27 16:53:06 +02:00
Todd Lipcon
7a3c6e9c3c HADOOP-15550. Avoid static initialization of ObjectMappers 2018-06-25 15:36:45 -07:00
Vidura Mudalige
1e94e5977f
MAPREDUCE-7063. Fix log level inconsistency in CombineFileInputFormat.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-06-18 14:25:11 -07:00
Arun Suresh
5670e89b2e MAPREDUCE-7101. Add config parameter to allow JHS to alway scan user dir irrespective of modTime. (Thomas Marquardt via asuresh) 2018-06-12 15:36:52 -07:00
Gera Shegalov
04b74eddc6 MAPREDUCE-7108. TestFileOutputCommitter fails on Windows. (Zuoming Zhang via gera) 2018-06-12 11:21:51 -07:00
Inigo Goiri
0afc036deb MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 2018-06-05 12:24:57 -07:00
Xiao Chen
6d5e87aec2 HADOOP-15507. Add MapReduce counters about EC bytes read. 2018-06-04 21:13:47 -07:00
Takanobu Asanuma
d1e2b80980
MAPREDUCE-7098. Upgrade common-langs version to 3.7 in hadoop-mapreduce-project 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-05-31 18:31:10 +09:00
Eric Yang
8cdb032aff YARN-8223. Improved yarn auxiliary service to load jar file from HDFS. 
Contributed by Zian Chen
 2018-05-04 12:36:31 -04:00
Bibin A Chundatt
1a95a4524a
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-05-02 16:14:28 +09:00
Jason Lowe
68c6ec719d MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin 2018-05-01 16:19:53 -05:00
Ewan Higgs
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-27 19:21:37 +09:00
Robert Kanter
1b9ecc264a MAPREDUCE-7072. mapred job -history prints duplicate counter in human output (wilfreds via rkanter) 2018-04-24 11:30:38 -07:00
Wangda Tan
995cba65fe MAPREDUCE-7077. Pipe mapreduce job fails with Permission denied for jobTokenPassword. (Akira Ajisaka via wangda) 
Change-Id: Ie8f01425d58409fa3661f768205b7616128c8aa4
(cherry picked from commit 035e0f97ea44b0495707949a781d8792dcf6ea6b)
 2018-04-13 13:52:10 -07:00
Jason Lowe
4571351ccc MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 2018-04-12 11:12:46 -05:00
Rohith Sharma K S
6cc59a09e7 MAPREDUCE-7062. Update mapreduce.job.tags description for making use for ATSv2 purpose. Contributed by Charan Hebri. 2018-04-11 09:45:39 +05:30
Steve Loughran
dde1579096 HADOOP-15146. Remove DataOutputByteBuffer. 
Contributed by BELUGA BEHR.
 2018-04-02 15:52:40 +01:00
Takanobu Asanuma
86816da5b4
MAPREDUCE-7066. TestQueue fails on Java9 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-19 09:39:49 +09:00
Jason Lowe
32fa3a63e0 MAPREDUCE-6930. mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml. Contributed by Sen Zhao 2018-03-09 10:41:16 -06:00
Weiwei Yang
6e6945cd78 MAPREDUCE-7059. Downward Compatibility issue: MR job fails because of unknown setErasureCodingPolicy method from 3.x client to HDFS 2.x cluster. Contributed by Jiandan Yang. 2018-03-01 10:18:53 +08:00
Billie Rinaldi
7dd385098c MAPREDUCE-7010. Make Job History File Permissions configurable. Contributed by Gergely Novák 2018-02-26 14:32:46 -08:00
Jason Lowe
87e2570a14 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko 2018-02-12 13:21:09 -06:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Eric Payne
5a725bb886 MAPREDUCE-7033: Map outputs implicitly rely on permissive umask for shuffle. Contributed by Jason Lowe 2018-01-31 15:45:30 -06:00
Takanobu Asanuma
56feaa40bb
MAPREDUCE-7036. ASF License warning in hadoop-mapreduce-client 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-30 00:39:39 +09:00
Jason Lowe
6eef3d7f1a MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko 2018-01-26 15:31:43 -06:00
Jason Lowe
a37e7f0ad8 MAPREDUCE-7022. Fast fail rogue jobs based on task scratch dir size. Contributed by Johan Gustavsson 2018-01-26 14:36:45 -06:00
Robert Kanter
836643d793 MAPREDUCE-6995. Uploader tool for Distributed Cache Deploy documentation (miklos.szegedi@cloudera.com via rkanter) 2018-01-19 17:57:54 -08:00
Jason Lowe
6e42d05829 MAPREDUCE-7029. FileOutputCommitter is slow on filesystems lacking recursive delete. Contributed by Karthik Palaniappan 2018-01-17 08:14:11 -06:00
Takanobu Asanuma
1a9c5d479e
MAPREDUCE-7034. Moving logging APIs over to slf4j the rest of all in hadoop-mapreduce 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:38:03 +09:00
Rohith Sharma K S
3cd75845da MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran. 2017-11-27 22:01:00 +05:30
Steve Loughran
de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
Chris Douglas
782ba3bf9d MAPREDUCE-7011. TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec 2017-11-21 20:42:28 -08:00
Naganarasimha
61bbdc511e MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph. 2017-11-06 12:58:38 +05:30
Akira Ajisaka
178751ed8c
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling. 2017-11-02 17:43:08 +09:00
Daniel Templeton
9a7e810838 YARN-6927. Add support for individual resource types requests in MapReduce 
(Contributed by Gergo Repas via Daniel Templeton)
 2017-10-30 11:04:22 -07:00
Akira Ajisaka
2523e1cce5
MAPREDUCE-6972. Enable try-with-resources for RecordReader. Contributed by Zoltan Haindrich. 2017-10-18 11:18:39 +09:00
Chris Trezzo
e46d5bb962 MAPREDUCE-5951. Add support for the YARN Shared Cache. 2017-10-12 10:59:20 -07:00
Jason Lowe
9df05005ac MAPREDUCE-6968. Staging directory erasure coding config property has a typo. Contributed by Jason Lowe 2017-09-26 14:52:27 -05:00
Robert Kanter
0adc0471d0 MAPREDUCE-6954. Disable erasure coding for files that are uploaded to the MR staging area (pbacsko via rkanter) 2017-09-18 10:40:06 -07:00
Robert Kanter
5f496683fb Revert "YARN-7162. Remove XML excludes file format (rkanter)" - wrong commit message 
This reverts commit 3a8d57a0a2.
 2017-09-18 10:32:08 -07:00
Robert Kanter
3a8d57a0a2 YARN-7162. Remove XML excludes file format (rkanter) 2017-09-15 12:00:26 -07:00
Steve Loughran
11390c2d11
MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter. 
Contributed by Steve Loughran
 2017-09-15 16:59:04 +01:00
Jason Lowe
4d98936eec MAPREDUCE-6957. shuffle hangs after a node manager connection timeout. Contributed by Jooseong Kim 2017-09-13 17:21:13 -05:00
Ray Chiang
556812c179 HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 2017-09-12 10:19:34 -07:00