big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,253 Commits 2 Branches 2 Tags 264 MiB
dea53dad67
Commit Graph

76 Commits

Author SHA1 Message Date
Steve Loughran
ab594ec77e
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-16 13:41:17 +01:00
Steve Loughran
e123de9f19
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
 2022-04-27 19:23:25 +01:00
Akira Ajisaka
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655)
 2021-02-22 23:16:05 +09:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Wanqiang Ji
eb6134cd22
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd4188)
 2020-09-29 23:07:58 +09:00
Wanqiang Ji
ffb96775e3
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86)
 2020-08-06 09:36:34 +09:00
Szilard Nemeth
ac6c4f0b29 MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal 2019-08-12 13:54:28 +02:00
Steve Loughran
9f1c017f44
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-21 11:15:37 +00:00
Shweta Yakkali
cb4d911a82 MAPREDUCE-7188. [Clean-up] Remove NULL check before instanceof and fix checkstyle issue in TaskResult 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: Ie5eb9462f94e45cfd9d2a4b85c081ac8be949c07
 2019-03-18 04:45:01 -07:00
Weiwei Yang
9e7196749e MAPREDUCE-7174. Mapreduce example wordmedian should handle generic options. Contributed by Fei Hui. 2019-01-02 21:39:45 +08:00
Takanobu Asanuma
4971276631
HADOOP-15271. Remove unicode multibyte characters from JavaDoc 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-05 15:07:58 -08:00
Jason Lowe
908d8e914e MAPREDUCE-6965. QuasiMonteCarlo should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 2017-09-22 09:34:53 -05:00
Akira Ajisaka
c71d137960
MAPREDUCE-6966. DistSum should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 2017-09-22 15:07:59 +09:00
Akira Ajisaka
53be075241
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergely Novák. 2017-09-22 13:27:59 +09:00
Jason Lowe
b9db0afa1e MAPREDUCE-6964. BaileyBorweinPlouffe should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 2017-09-21 09:37:19 -05:00
Akira Ajisaka
53047f934e
Revert "MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák." 
This reverts commit 2018538fdb.
 2017-09-21 11:16:05 +09:00
Andrew Wang
51edaacd09 HDFS-12445. Correct spellings of choosen to chosen. Contributed by hu xiaodong. 2017-09-19 13:48:23 -07:00
Akira Ajisaka
2018538fdb
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák. 2017-09-19 11:05:54 +09:00
Anu Engineer
c35510a465 HADOOP-14849. some wrong spelling words update. Contributed by Chen Hongfei. 2017-09-08 10:02:34 -07:00
Arpit Agarwal
8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Akira Ajisaka
d4015f8628
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin. 2017-05-30 14:48:58 +09:00
Sunil G
b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Akira Ajisaka
3ed3062fe3
MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. Contributed by Weiwei Yang. 2017-04-27 15:45:33 +09:00
Akira Ajisaka
490abfb10f HADOOP-14057. Fix package.html to compile with Java 9. 2017-03-04 00:25:22 +09:00
Daniel Templeton
deb368b1b6 MAPREDUCE-6842. Update the links in PiEstimator document (Contributed by Jung Yoo via Daniel Templeton) 2017-02-07 13:05:40 -08:00
Daniel Templeton
0b8a7c18dd MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu via Daniel Templeton) 2017-01-05 17:56:08 -08:00
Mingliang Liu
5af572b644 HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 2016-11-15 10:57:00 -08:00
Kai Zheng
bea004eaeb MAPREDUCE-6780. Add support for HDFS directory with erasure code policy to TeraGen and TeraSort. Contributed by Sammi Chen 2016-10-09 15:33:26 +06:00
Wei-Chiu Chuang
1360bd2d54 MAPREDUCE-6764. Teragen LOG initialization bug. Contributed by Yufei Gu. 2016-08-25 09:50:12 -07:00
Chris Nauroth
26ac0f8f0b MAPREDUCE-6672. TestTeraSort fails on Windows. Contributed by Tibor Kiss. 2016-04-29 11:33:32 -07:00
Akira Ajisaka
80fa70c4e1 MAPREDUCE-6645. TestWordStats outputs logs under directories other than target/test-dir. Contributed by Gabor Liptak. 2016-03-17 18:08:31 +09:00
Akira Ajisaka
247a790609 MAPREDUCE-6520. Migrate MR Client test cases part 1. Contributed by Dustin Cote. 2016-03-11 22:51:20 +09:00
Akira Ajisaka
6876b9f9f9 Revert "MAPREDUCE-6520. Migrate MR Client test cases part 1." 
This reverts commit ef68b44195.
 2016-03-11 22:50:57 +09:00
Akira Ajisaka
ef68b44195 MAPREDUCE-6520. Migrate MR Client test cases part 1. 2016-03-11 22:20:55 +09:00
Varun Vasudev
2440671a11 MAPREDUCE-6634. Log uncaught exceptions/errors in various thread pools in mapreduce. Contributed by Sidharta Seethana. 2016-02-18 14:18:36 +05:30
Devaraj K
c7729efee8 MAPREDUCE-6389. Fix BaileyBorweinPlouffe CLI usage message. Contributed by 
Brahma Reddy Battula.
 2015-06-10 21:05:10 +05:30
Devaraj K
09b8241e6e MAPREDUCE-6383. Pi job (QuasiMonteCarlo) should not try to read the 
results file if its job fails. Contributed by Harsh J.
 2015-06-05 21:22:47 +05:30
Tsuyoshi Ozawa
e82067bfe6 MAPREDUCE-6366. mapreduce.terasort.final.sync configuration in TeraSort doesn't work. Contributed by Takuya Fukudome. 2015-05-13 16:44:50 +09:00
Harsh J
b375d1fc93 MAPREDUCE-6287. Deprecated methods in org.apache.hadoop.examples.Sort. Contributed by Chao Zhang. 2015-03-23 03:48:36 +05:30
Tsuyoshi Ozawa
1ccbc29708 MAPREDUCE-6281. Fix javadoc in Terasort. Contributed by Albert Chu. 2015-03-20 00:58:34 +09:00
Harsh J
9d72f93975 MAPREDUCE-5807. Print usage for TeraSort job. Contributed by Rohith. 2015-03-18 15:36:52 +05:30
Tsuyoshi Ozawa
9ae7f9eb7b MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA. 2015-03-03 18:06:26 +09:00
Tsuyoshi Ozawa
d1c6accb6f HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-03-03 14:17:52 +09:00
Tsuyoshi Ozawa
9cedad11d8 Revert "HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)" 
This reverts commit 946456c6d8.

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
 2015-02-25 00:32:04 +09:00
Tsuyoshi Ozawa
946456c6d8 HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-02-19 13:06:53 +09:00
Akira Ajisaka
bd69fb2d44 MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors. (aajisaka) 2015-02-03 14:30:09 -08:00
Akira Ajisaka
fe2188a042 MAPREDUCE-6231. Grep example job is not working on a fully-distributed cluster. (aajisaka) 2015-01-30 01:10:02 +09:00
Gera Shegalov
25a0440238 MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output directory. (gera) 2014-12-13 17:52:50 -08:00
Haohui Mai
be86237c09 HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. Contributed by Li Lu. 2014-12-09 10:49:55 -08:00
Ivan Mitic
da80c4da41 Revert "MAPREDUCE-5911. Terasort TeraOutputFormat does not check for output directory existance. Contributed by Bruno P. Kinoshita." 
This reverts commit 7bbda6ef92.
 2014-10-19 23:57:12 -04:00