big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,674 Commits 2 Branches 2 Tags 264 MiB
2ece95064b
Commit Graph

910 Commits

Author SHA1 Message Date
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Viraj Jasani
08c803ea30
MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally (#3855) 2022-01-09 00:18:10 +09:00
Stamatis Zampetakis
bface2ac6c
MAPREDUCE-7368. DBOutputFormat.DBRecordWriter#write must throw exception when it fails. (#3671). Contributed by Stamatis Zampetakis. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-08 16:40:11 +05:30
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
Viraj Jasani
b1ad4eab9a
HADOOP-17959. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules (#3537) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 16:22:50 +09:00
Chao Sun
2ee294b1b1 Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko." 
This reverts commit 7bc305db5d.
 2021-09-25 09:29:33 -07:00
lzx404243
6187f76f11
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
 2021-09-01 13:47:09 +09:00
Viraj Jasani
e95c3259de
MAPREDUCE-7356. Remove some duplicate dependencies from mapreduce-client's child poms (#3193). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-13 19:30:13 +05:30
Viraj Jasani
6e11461eaa
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-21 16:46:06 +09:00
Viraj Jasani
207c92753f
MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project (#3074) 2021-06-07 11:51:29 +09:00
Viraj Jasani
2a206c20cc
HADOOP-17722. Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project (#3032) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:07:22 +09:00
Ayush Saxena
5404ab4bca
MAPREDUCE-7343. Increase the job name max length in mapred job -list. (#2995). Contributed by Ayush Saxena. 2021-05-14 00:15:33 +05:30
Akira Ajisaka
1448756505
MAPREDUCE-7334. TestJobEndNotifier fails. (#2877) 2021-04-08 23:53:51 -07:00
lichaojacobs
663ca14a76
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x (#2775) 2021-04-09 11:58:53 +09:00
Sungpeo Kook
46a5979805
MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English. (#1942) 
Contributed by Sungpeo Kook.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-07 23:58:10 -07:00
Jim Brennan
ede490d131 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 2021-03-22 18:41:25 +00:00
Jim Brennan
299b8062f1 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-15 20:13:17 +00:00
Jungtaek Lim
2a38ed0e0c
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624) 
Contributed by Jungtaek Lim.
 2021-01-27 19:08:08 +00:00
Steve Loughran
9b2956e254
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579) 
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null. 

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.
 2020-12-31 16:02:10 +00:00
dengzh
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469) 
Contributed by Zhihua Deng.
 2020-11-23 15:40:22 +00:00
Peter Bacsko
fb92aa4012 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:13:51 +01:00
Peter Bacsko
da93cd962c MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 2020-10-28 10:35:30 +01:00
Akira Ajisaka
7bc305db5d
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 2020-10-23 04:34:49 +09:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
zz
95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
ywheel
cf4eb75608
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 2020-07-30 13:01:22 +09:00
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Eric Badger
fbb8775430 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit e2322e1117.
 2020-06-08 20:35:27 +00:00
Eric E Payne
e2322e1117 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 2020-04-27 19:10:00 +00:00
Surendra Singh Lilhore
a1b0697d37 MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T 2020-04-18 19:42:20 +05:30
Eric E Payne
11d17417ce MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein) 2020-04-13 18:20:07 +00:00
Jason Lowe
c613296dc8 MAPREDUCE-7241. FileInputFormat listStatus with less memory footprint. Contributed by Zhihua Deng 2020-04-01 07:46:33 -05:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Wanqiang Ji
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters (#1397) 2020-03-16 11:28:36 +09:00
Sergey Pogorelov
b343e1533b MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793) 2020-01-06 12:36:11 +09:00
Ahmed Hussein
ed302f1fed MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 14:55:20 -06:00
Steve Loughran
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3 
Contributed by Steve Loughran.

Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.

This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.

Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
 2019-10-01 18:11:05 +01:00
Daisuke Kobayashi
bc2d3a71d6 HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-10 10:51:47 +08:00
Szilard Nemeth
ac6c4f0b29 MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal 2019-08-12 13:54:28 +02:00
Szilard Nemeth
a7371a779c MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 13:01:30 +02:00
Mehul Garnara
c0a0c353e8
MAPREDUCE-6973. Fix comments on creating _SUCCESS file. 
This closes #280

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-26 21:21:26 +09:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Wanqiang Ji
e7e30a5f8b
MAPREDUCE-7210. Replace mapreduce.job.counters.limit with mapreduce.job.counters.max in mapred-default.xml 
This closes #878

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-05 10:45:23 +09:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Akira Ajisaka
5565f2c532
MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work. 2019-05-23 10:21:11 +09:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
David Mollitor
246ab77f28
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>
 2019-03-22 10:26:24 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00