big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,662 Commits 2 Branches 2 Tags 264 MiB
78723e045a
Commit Graph

2902 Commits

Author SHA1 Message Date
Wilfred Spiegelenburg
586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false 
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c)
 2020-05-28 13:36:21 +10:00
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
Szilard Nemeth
e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Eric E Payne
741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117)
 2020-04-27 19:34:38 +00:00
Eric E Payne
b397a3a875 MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit 11d17417ce)
 2020-04-13 18:51:00 +00:00
Szilard Nemeth
ef01e9d40e MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja 
(cherry picked from commit e2c87b7917)
 2020-04-09 12:40:22 +02:00
Akira Ajisaka
cb806221e9
MAPREDUCE-7269. TestNetworkedJob fails (#1940) 
(cherry picked from commit ab7495dc7d)
 2020-04-06 17:58:25 +09:00
Wanqiang Ji
215a62e99f
MAPREDUCE-7268. Fix TestMapreduceConfigFields (#1935) 
(cherry picked from commit f8e598f278)
 2020-04-03 11:00:02 +09:00
Wanqiang Ji
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters (#1397) 2020-03-16 11:28:36 +09:00
Szilard Nemeth
8d6ff87c18 MAPREDUCE-7263. Remove obsolete validateTargetPath() from FrameworkUploader. Contributed by Marton Hudaky 2020-02-12 15:53:33 +01:00
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth
71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Eric E Payne
b897f6834b MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 2020-01-29 16:54:45 +00:00
Szilard Nemeth
7f3e1e0c07 MAPREDUCE-7260. Cross origin request support for Job history server web UI. Contributed by Adam Antal 2020-01-29 14:42:52 +01:00
Ahmed Hussein
08251538fe MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-28 10:57:33 -06:00
Ahmed Hussein
3f01c48106 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-27 15:50:13 -06:00
Szilard Nemeth
9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
kevinzhao1661
b8518e6dab MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int (#1684) 2020-01-16 12:53:56 +09:00
Masatake Iwasaki
a2fdd7c2b5 MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804) 2020-01-15 12:09:09 +09:00
Szilard Nemeth
13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00
Eric E Payne
93233a7d6e YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 2020-01-09 17:18:44 +00:00
Ahmed Hussein
cdd6efd3ab MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-01-08 11:08:13 -06:00
Sergey Pogorelov
b343e1533b MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793) 2020-01-06 12:36:11 +09:00
Sunil G
682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
prabhujoseph
da9aa9c909 MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0. 
Contributed by Peter Bacsko.
 2019-12-05 12:32:57 +05:30
prabhujoseph
304e75a48b Revert "MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0." 
This reverts commit 8d82c44102.
 2019-12-05 12:31:13 +05:30
prabhujoseph
8d82c44102 MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0. 
Contributed by Peter Bacsko.
 2019-12-04 20:20:56 +05:30
prabhujoseph
013c6e51b6 MAPREDUCE-7249. Fix Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure. 
Contributed by Wilfred Spiegelenburg.
 2019-11-28 10:57:15 +05:30
prabhujoseph
7f2ea2ac46 MAPREDUCE-7240. Fix Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER. 
Contributed by Huachao and Peter Bacsko. Reviewed by Wilfred Spiegelenburg.
 2019-11-27 14:27:27 +05:30
Ahmed Hussein
ed302f1fed MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 14:55:20 -06:00
Jonathan Hung
e6137d0e50 Make upstream aware of 2.10.0 release 2019-10-30 20:58:32 -07:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Steve Loughran
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3 
Contributed by Steve Loughran.

Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
 2019-10-01 18:11:05 +01:00
Duo Zhang
e8e7d7b4cd
HADOOP-16561. [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes (#1500) 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:52:12 +08:00
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Tao Yang
5d497abe21 YARN-9817. Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero. Contributed by Prabhu Joseph. 2019-09-07 07:39:02 +08:00
Szilard Nemeth
3f4f097f1f MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph 2019-08-15 17:07:43 +02:00
Szilard Nemeth
ac6c4f0b29 MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal 2019-08-12 13:54:28 +02:00
Szilard Nemeth
a7371a779c MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 13:01:30 +02:00
Chen Zhang
8a59cd1b8a HDFS-12967. NNBench should support multi-cluster access. Contributed by Chen Zhang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-29 05:55:12 -07:00
Mehul Garnara
c0a0c353e8
MAPREDUCE-6973. Fix comments on creating _SUCCESS file. 
This closes #280

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-26 21:21:26 +09:00
pingsutw
ee87e9a42e
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build 
This closes #1089

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-23 10:34:10 +09:00
Masatake Iwasaki
79f6118dcc
MAPREDUCE-6521. MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test. 
Contributed by Masatake Iwasaki.

Change-Id: Id74b90eb7cfb9d676188fc5aa47249ee8904a3d5
 2019-07-18 13:15:18 +01:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Wanqiang Ji
e7e30a5f8b
MAPREDUCE-7210. Replace mapreduce.job.counters.limit with mapreduce.job.counters.max in mapred-default.xml 
This closes #878

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-05 10:45:23 +09:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Akira Ajisaka
5565f2c532
MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work. 2019-05-23 10:21:11 +09:00
Wanqiang Ji
67f9a7b165
MAPREDUCE-7205. Treat container scheduler kill exit code as a task attempt killing event. 
This closes #821

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-05-22 14:23:13 +09:00
Akira Ajisaka
afe6613ee6
MAPREDUCE-7200. Remove stale eclipse templates. Contributed by Wanqiang Ji. 2019-04-25 11:18:39 +09:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Adam Antal
856cbf62d3 MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-01 10:25:38 -07:00
David Mollitor
246ab77f28
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>
 2019-03-22 10:26:24 +00:00
Steve Loughran
9f1c017f44
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-21 11:15:37 +00:00
Shweta Yakkali
cb4d911a82 MAPREDUCE-7188. [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: Ie5eb9462f94e45cfd9d2a4b85c081ac8be949c07
 2019-03-18 04:45:01 -07:00
Weiwei Yang
159a715eef MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang. 2019-03-13 17:00:35 +08:00
Weiwei Yang
f0605146b3 MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang. 2019-03-11 15:02:54 +08:00
Wangda Tan
308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Akira Ajisaka
c991e2ca5a
MAPREDUCE-7177. Disable speculative execution in TestDFSIO. Contributed by Zhaohui Xin. 2019-02-04 14:03:34 +09:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Eric Yang
1ab69a9543 YARN-9221. Added flag to disable dynamic auxiliary service feature. 
Contributed by Billie Rinaldi
 2019-01-25 19:05:36 -05:00
Eric Yang
a33ef4fd31 YARN-8867. Added resource localization status to YARN service status call. 
Contributed by Chandni Singh
 2019-01-24 18:43:21 -05:00
Eric Yang
2fa9389c2e YARN-9146. Added REST API to configure auxiliary service. 
Contributed by Billie Rinaldi
 2019-01-22 18:24:43 -05:00
Steve Loughran
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran and Daryn Sharp.
 2019-01-14 17:59:27 +00:00
Akira Ajisaka
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens." 
This reverts commit d7152332b3.
 2019-01-08 14:51:30 +09:00
Steve Loughran
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran.
 2019-01-07 13:18:03 +00:00
Weiwei Yang
9e7196749e MAPREDUCE-7174. Mapreduce example wordmedian should handle generic options. Contributed by Fei Hui. 2019-01-02 21:39:45 +08:00
Eric Yang
f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Eric Yang
7affa3053c YARN-9131. Updated document usage for dynamic auxiliary service. 
Contributed by Billie Rinaldi
 2018-12-20 18:45:49 -05:00
Akira Ajisaka
499c70eda5
MAPREDUCE-7166. map-only job should ignore node lost event when task is already succeeded. Contributed by Lei Li. 2018-12-20 10:09:50 +09:00
Eric Yang
c7a5a4435e YARN-9075. Add ability to register/remove auxiliary service without restart node manager. 
Contributed by Billie Rinaldi
 2018-12-18 17:05:51 -05:00
Akira Ajisaka
e0d75088f2
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin. 2018-12-13 16:26:25 +09:00
Haibo Chen
881230da21 YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth) 2018-12-11 11:41:32 -08:00
Jason Lowe
9886c27c71 MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used. Contributed by Peter Bacsko 2018-12-06 14:48:17 -06:00
Jason Lowe
4d8de7ab69 MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla 2018-11-28 14:54:59 -06:00
Akira Ajisaka
13a21f6607
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin. 2018-11-28 17:57:42 +09:00
Akira Ajisaka
1aad99a718
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin. 2018-11-27 20:01:11 +09:00
Billie Rinaldi
27ffec7ba7 YARN-8778. Add command line interface to invoke interactive docker shell. Contributed by Eric Yang 2018-11-19 17:59:12 -08:00
Arpit Agarwal
1e15c7e855 HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel. 2018-11-15 12:42:31 -08:00
Wangda Tan
762a56cc64 MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda) 
Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a
 2018-11-13 11:25:41 -08:00
Jason Lowe
0b6625a973 MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan 2018-11-07 08:20:49 -06:00
Jason Lowe
ba1f9d66d9 MAPREDUCE-7156. NullPointerException when reaching max shuffle connections. Contributed by Peter Bacsko 2018-11-06 17:55:51 -06:00
Jason Lowe
ab6aa4c726 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 2018-11-06 08:40:59 -06:00
bibinchundatt
74a5e683fe MAPREDUCE-7151. RMContainerAllocator#handleJobPriorityChange expects application_priority always. Contributed by Bilwa S T. 2018-10-24 23:22:05 +05:30
Haibo Chen
823bb5dda8 MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter) 2018-10-23 15:28:13 -07:00
Haibo Chen
babd1449bf MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev) 2018-10-16 13:44:41 -07:00
Xiao Chen
25f8fcb064 MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko. 2018-10-16 10:23:31 -07:00
Steve Loughran
735a36afd5
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection. 
Contributed by Steve Loughran.
 2018-10-11 20:01:42 +01:00
Akira Ajisaka
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter. 2018-10-10 10:16:57 +09:00
Akira Ajisaka
5b1cfcaeff
MAPREDUCE-7035. Skip javadoc build for auto-generated sources in hadoop-mapreduce-client. Contributed by Mukul Kumar Singh. 2018-10-09 21:13:04 +09:00
Akira Ajisaka
31fa9bb799
HADOOP-15767. [JDK10] Building native package on JDK10 fails due to missing javah. Contributed by Takanobu Asanuma. 2018-10-03 11:29:18 +09:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Wangda Tan
230d90c5b9 MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda) 
Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a
 2018-09-24 13:52:53 -07:00
Jason Lowe
8382b860d4 MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler. Contributed by Oleksandr Shevchenko 2018-09-18 17:06:32 -05:00
Jason Lowe
34b2237e42 MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with NullPointerException. Contributed by Oleksandr Shevchenko 2018-09-18 16:56:31 -05:00
Eric E Payne
2cf8927683 MAPREDUCE-3801: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently. Contributed by Jason Lowe 2018-09-18 20:26:28 +00:00
Jason Lowe
488806baca MAPREDUCE-7140. Refactoring TaskAttemptInfo to separate Map and Reduce tasks. Contributed by Oleksandr Shevchenko 2018-09-14 15:10:27 -05:00