big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
15,487 Commits 2 Branches 2 Tags 264 MiB
82ef9accaf
Commit Graph

508 Commits

Author SHA1 Message Date
Sangjin Lee
605b4b6136 MAPREDUCE-6719. The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store (Daniel Templeton via sjlee) 2016-06-21 11:25:11 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Akira Ajisaka
e3ba9ad3f1 HADOOP-13270. BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks. Contributed by Kai Sasaki. 2016-06-14 10:18:41 +09:00
Gera Shegalov
0af96a1c08 MAPREDUCE-6240. Hadoop client displays confusing error message. (gera) 2016-06-08 12:59:37 -07:00
Robert Kanter
733f3f18d5 MAPREDUCE-6702. TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are failing (ajisakaa via rkanter) 2016-06-07 15:46:06 -07:00
Ming Ma
4a1cedc010 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma) 2016-06-06 14:30:51 -07:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Carlo Curino
013532a95e YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino) 2016-05-25 16:55:49 -07:00
Jian He
ae353ea969 MAPREDUCE-6703. Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers. Contributed by Arun Suresh 2016-05-24 19:47:50 -07:00
Jian He
21d2b90213 MAPREDUCE-6696. Add a configuration to limit the number of map tasks allowed per job. Contributed by Zhihai Xu 2016-05-19 19:25:16 -07:00
Robert Kanter
992a49353f MAPREDUCE-6686. Add a way to download the job config from the mapred CLI (rkanter) 2016-05-18 12:16:21 -07:00
Robert Kanter
ecce3b7d53 MAPREDUCE-6677. LocalContainerAllocator doesn't specify resource of the containers allocated (haibochen via rkanter) 2016-05-04 22:19:20 -07:00
Harsh J
7da5847cf1 MAPREDUCE-2398. MRBench: setting the baseDir parameter has no effect. Contributed by Michael Noll and Wilfred Spiegelenburg. 2016-04-21 17:53:51 +05:30
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Akira Ajisaka
690d8a368d MAPREDUCE-6663. [NNBench] Refactor nnbench as a Tool implementation. Contributed by Brahma Reddy Battula. 2016-03-30 11:42:54 +09:00
Akira Ajisaka
0050fa5f1c MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-29 18:17:52 +09:00
Rohith Sharma K S
0ef8bbfd87 MAPREDUCE-6662. Clear ASF Warnings on test data files. Contributed by Vinayakumar B 2016-03-29 10:32:21 +08:00
Akira Ajisaka
115be193df Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 2c268cc936.
 2016-03-28 16:05:35 +09:00
Akira Ajisaka
2c268cc936 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 2016-03-25 15:17:27 +09:00
Akira Ajisaka
368c77376a MAPREDUCE-6656. [NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363. Contributed by J.Andreina. 2016-03-23 23:22:43 +09:00
Akira Ajisaka
0bfe5a0167 MAPREDUCE-6580. Test failure: TestMRJobsWithProfiler. Contributed by Eric Badger. 2016-03-23 13:18:06 +09:00
Wangda Tan
6529c87551 MAPREDUCE-6579. JobStatus#getFailureInfo should not output diagnostic information when the job is running. (Akira AJISAKA via wangda) 2016-03-15 17:28:40 -07:00
Akira Ajisaka
247a790609 MAPREDUCE-6520. Migrate MR Client test cases part 1. Contributed by Dustin Cote. 2016-03-11 22:51:20 +09:00
Akira Ajisaka
6876b9f9f9 Revert "MAPREDUCE-6520. Migrate MR Client test cases part 1." 
This reverts commit ef68b44195.
 2016-03-11 22:50:57 +09:00
Akira Ajisaka
ef68b44195 MAPREDUCE-6520. Migrate MR Client test cases part 1. 2016-03-11 22:20:55 +09:00
Robert Kanter
4343a4cf77 MAPREDUCE-6640. mapred job -history command should be able to take Job ID (rkanter) 2016-02-23 17:37:49 -08:00
Tsuyoshi Ozawa
b68901d7dd YARN-4708. Missing default mapper type in TimelineServer performance test tool usage. Contributed by Kai Sasaki. 2016-02-22 07:54:53 +09:00
Robert Kanter
8eee59ce6b MAPREDUCE-6627. Add machine-readable output to mapred job -history command (rkanter) 2016-02-18 17:58:26 -08:00
Wangda Tan
9875325d5c YARN-4340. Add list API to reservation system. (Sean Po via wangda) 2016-02-02 10:17:33 +08:00
Jason Lowe
59a212b6e1 MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient thread. Contributed by Xuan Gong 2016-02-01 16:05:06 +00:00
Karthik Kambatla
d40859fab1 YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha) 2016-01-18 10:58:14 +01:00
Akira Ajisaka
7b0964f354 MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Vlad Sharanhovich and Bibin A Chundatt. 2016-01-14 10:40:22 +09:00
Sangjin Lee
f6f16118d3 MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee) 2016-01-05 15:22:50 -08:00
Jian He
85c2466048 YARN-4164. Changed updateApplicationPriority API to return the updated application priority. Contributed by Rohith Sharma K S 2015-12-18 14:13:48 -08:00
Jason Lowe
f634505d48 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G 2015-11-24 22:07:26 +00:00
Haohui Mai
000e12f6fa HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai Zheng. 2015-11-22 16:18:07 -08:00
Sangjin Lee
9ad708a0df MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee) 2015-11-11 15:37:24 -08:00
Akira Ajisaka
04d97f8abb MAPREDUCE-6525. Fix test failure of TestMiniMRClientCluster.testRestart. Contributed by Masatake Iwasaki. 2015-11-03 01:50:07 +09:00
Xuan
7412ff48ee YARN-4313. Race condition in MiniMRYarnCluster when getting history 
server address. Contributed by Jian He
 2015-10-29 17:36:36 -07:00
Tsuyoshi Ozawa
96677bef00 YARN-3573. MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked deprecated. Contributed by Brahma Reddy Battula. 2015-10-27 12:35:01 +09:00
Junping Du
eb6379ca25 MAPREDUCE-6508. TestNetworkedJob fails consistently due to delegation token changes on RM. Contributed by Akira AJISAKA 2015-10-23 10:05:46 -07:00
Sangjin Lee
58590fef49 YARN-2556. Tool to measure the performance of the timeline server (Chang Li via sjlee) 2015-10-16 16:53:20 -07:00
Jason Lowe
a0bca2b5ad YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S 2015-10-09 14:17:38 +00:00
Xuan
8f08532bde YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma 2015-10-02 18:50:47 -07:00
Haohui Mai
602335dfe6 MAPREDUCE-6483. Replace deprecated method NameNode.getUri() with DFSUtilClient.getNNUri() in TestMRCredentials. Contributed by Mingliang Liu. 2015-09-18 13:36:09 -07:00
Haohui Mai
941001bfa6 MAPREDUCE-6477. Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem. Contributed by Mingliang Liu. 2015-09-16 13:06:26 -07:00
Jason Lowe
62943b8e3a MAPREDUCE-6472. MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks. Contributed by Naganarasimha G R 2015-09-15 19:43:20 +00:00
Andrew Wang
e1feaf6db0 HADOOP-12358. Add -safely flag to rm to prompt when deleting many files. Contributed by Xiaoyu Yao. 2015-09-04 13:42:55 -07:00
Akira Ajisaka
00804e2457 Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." 
This reverts commit 75a2560e51.
 2015-09-02 12:22:16 +09:00
Zhihai Xu
cbb249534a MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu 2015-08-28 12:13:23 -07:00
Jian He
57c7ae1aff YARN-4014. Support user cli interface in for Application Priority. Contributed by Rohith Sharma K S 2015-08-24 20:36:44 -07:00
Jason Lowe
aa5b15b03b MAPREDUCE-6413. TestLocalJobSubmission is failing with unknown host. Contributed by zhihai xu 2015-06-25 19:50:07 +00:00
Akira Ajisaka
445b13240c MAPREDUCE-6403. Fix typo in the usage of NNBench. Contributed by Jagadesh Kiran N. 2015-06-22 15:32:39 +09:00
Akira Ajisaka
85cc644f92 MAPREDUCE-6396. TestPipeApplication fails by NullPointerException. Contributed by Brahma Reddy Battula. 2015-06-15 15:28:31 -07:00
Akira Ajisaka
75a2560e51 MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. 2015-06-15 14:52:26 -07:00
Zhijie Shen
bc85959edd Revert "YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong." 
This reverts commit 4a9ec1a824.
 2015-06-03 14:15:56 -07:00
Zhijie Shen
4a9ec1a824 YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong. 2015-05-30 21:05:36 -07:00
Wangda Tan
3164e7d838 MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) 2015-05-27 14:26:03 -07:00
Tsuyoshi Ozawa
b27ce3c531 MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS. 2015-05-22 23:29:53 +09:00
Wangda Tan
563eb1ad2a YARN-3583. Support of NodeLabel object instead of plain String in YarnClient side. (Sunil G via wangda) 2015-05-19 16:54:38 -07:00
Vinod Kumar Vavilapalli
6b710a42e0 Fixing MR intermediate spills. Contributed by Arun Suresh. 2015-05-14 16:07:56 -07:00
Vinod Kumar Vavilapalli
f24452d14e MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. 2015-05-12 12:11:42 -07:00
Jason Lowe
444836b3dc MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma 2015-05-11 22:37:35 +00:00
Ravi Prakash
5aab014340 MAPREDUCE-4750. Enable NNBenchWithoutMR in MapredTestDriver (Liang Xie and Jason Lowe via raviprak) 2015-05-08 15:56:29 -07:00
Jason Lowe
30099a36c6 MAPREDUCE-5248. Let NNBenchWithoutMR specify the replication factor for its test. Contributed by Erik Paulson 2015-05-08 22:29:47 +00:00
Akira Ajisaka
d335071398 MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 2015-05-07 19:39:24 +09:00
Tsuyoshi Ozawa
551615fa13 MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. 2015-05-05 10:23:13 +09:00
Akira Ajisaka
ac281e3fc8 HADOOP-11627. Remove io.native.lib.available. Contributed by Brahma Reddy Battula. 2015-04-24 08:08:55 +09:00
Vinod Kumar Vavilapalli
f5fe35e297 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv) 2015-04-23 11:19:55 -07:00
Gera Shegalov
725eb52ddc MAPREDUCE-6293. Set job classloader on uber-job's LocalContainerLauncher event thread. (Sangjin Lee via gera) 2015-04-21 13:57:22 -07:00
Robert Kanter
d50e8f0928 MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 2015-04-20 14:14:08 -07:00
Jian He
577d755e4b YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 09:57:17 -07:00
Harsh J
c91b2de68a MAPREDUCE-6314. TestPipeApplication fails on trunk. Contributed by Varun Vasudev. 2015-04-10 14:02:26 +05:30
Hitesh Shah
265ed1fe80 YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 2015-04-08 14:13:10 -07:00
Akira Ajisaka
c770df49b4 MAPREDUCE-6292. Use org.junit package instead of junit.framework in TestCombineFileInputFormat. (aajisaka) 2015-03-25 19:00:35 +09:00
Tsuyoshi Ozawa
4170c99147 MAPREDUCE-6285. ClientServiceDelegate should not retry upon AuthenticationException. Contributed by Jonathan Eagles. 2015-03-25 00:56:26 +09:00
Harsh J
7a678db3ac MAPREDUCE-6239. Consolidate TestJobConf classes in hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core. Contributed by Varun Saxena. 2015-03-21 09:43:29 +05:30
Akira Ajisaka
8041267f02 MAPREDUCE-4424. 'mapred job -list' command should show the job name as well. Contributed by Avinash Kujur. 2015-03-20 18:46:51 +09:00
Jonathan Eagles
8180e676ab YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 2015-03-13 12:04:30 -05:00
Tsuyoshi Ozawa
20b8ee1350 MAPREDUCE-4742. Fix typo in nnbench#displayUsage. Contributed by Liang Xie. 2015-03-11 00:41:12 +09:00
Junping Du
d39bc903a0 Configurable timeout between YARNRunner terminate the application and forcefully kill. Contributed by Eric Payne. 2015-03-10 06:21:59 -07:00
Tsuyoshi Ozawa
9ae7f9eb7b MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA. 2015-03-03 18:06:26 +09:00
Tsuyoshi Ozawa
d1c6accb6f HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-03-03 14:17:52 +09:00
Karthik Kambatla
1047c883ba MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. (Varun Saxena via kasha) 2015-02-26 14:24:19 -08:00
Tsuyoshi Ozawa
9cedad11d8 Revert "HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)" 
This reverts commit 946456c6d8.

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
 2015-02-25 00:32:04 +09:00
Wangda Tan
d49ae725d5 YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda) 2015-02-19 11:00:57 -08:00
Plamen Jeliazkov
a19820f2fb MAPREDUCE-6228. Add truncate operation to SLive. Constributed by Plamen Jeliazkov. 2015-02-19 00:02:49 -08:00
Tsuyoshi Ozawa
946456c6d8 HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-02-19 13:06:53 +09:00
Tsuyoshi Ozawa
7c782047c6 MAPREDUCE-4286. TestClientProtocolProviderImpls passes on failure conditions. Contributed by Devaraj K. 2015-02-18 15:45:52 +09:00
yliu
ef01768333 MAPREDUCE-6227. DFSIO for truncate. (shv via yliu) 2015-02-08 02:43:43 +08:00
Robert Kanter
e2ee2ff7d7 MAPREDUCE-6233. org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort failed in trunk (zxu via rkanter) 2015-02-05 14:20:28 -08:00
Akira Ajisaka
bd69fb2d44 MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors. (aajisaka) 2015-02-03 14:30:09 -08:00
Brandon Li
09d31bc630 HADOOP-9992. Modify the NN loadGenerator to optionally run as a MapReduce job. Contributed by Akshay Radia 2015-01-09 17:24:22 -08:00
Robert Kanter
390a7c12f5 MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter) 2014-12-19 15:43:48 -08:00
Robert Kanter
d9e4d67d18 MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter) 2014-12-19 11:59:22 -08:00
Eric Yang
c379e102dd MAPREDUCE-6191. Improve clearing stale state of Java serialization 
testcase.  (Sam Liu via Eric Yang)
 2014-12-15 19:39:41 -08:00
Jian He
a4f2995b9e Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-12-15 11:04:09 -08:00
Harsh J
cb99f43305 MAPREDUCE-5420. Remove mapreduce.task.tmp.dir from mapred-default.xml. Contributed by James Carman. (harsh) 2014-12-11 09:45:49 +05:30
Jason Lowe
03ab24aa01 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 2014-12-03 17:02:14 +00:00
Zhijie Shen
51af8d367d YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 2014-12-01 19:04:04 -08:00