big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19,957 Commits 2 Branches 2 Tags 264 MiB
0a01b1350d
Commit Graph

66 Commits

Author SHA1 Message Date
Haibo Chen
274eee3284 MAPREDUCE-7095. Race conditions in closing FadvisedChunkedFile. (Miklos Szegedi via Haibo Chen) 2018-05-10 10:46:55 -07:00
Eric E Payne
6bb128dfb8 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 2018-04-12 16:04:23 +00:00
Akira Ajisaka
e4c220ee4f
MAPREDUCE-7001. Moving logging APIs over to slf4j in hadoop-mapreduce-client-shuffle. Contributed by Jinjiang Ling. 2017-11-08 19:28:08 +09:00
Jason Lowe
665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Eric Payne
595d478408 MAPREDUCE-6960. Shuffle Handler prints disk error stack traces for every read failure. 2017-09-19 10:35:15 -05:00
Jason Lowe
3a20debdde MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe 2017-09-19 09:13:17 -05:00
Jason Lowe
ea845ba58c Revert "MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe" 
This reverts commit b3d61304f2.
 2017-09-19 08:45:05 -05:00
Jason Lowe
b3d61304f2 MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe 2017-09-18 17:04:43 -05:00
Ravi Prakash
ac7d0604bc MAPREDUCE-6923. Optimize MapReduce Shuffle I/O for small partitions. Contributed by Robert Schmidtke. 2017-08-09 15:39:52 -07:00
Akira Ajisaka
cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Jason Lowe
c8bd5fc7a8 MAPREDUCE-6850. Shuffle Handler keep-alive connections are closed from the server side. Contributed by Jonathan Eagles 2017-03-30 10:57:19 -05:00
Daniel Templeton
a7463b6c88 MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton) 2017-01-25 14:32:40 -08:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Robert Kanter
d4725bfcb2 MAPREDUCE-6728. Give fetchers hint when ShuffleHandler rejects a shuffling connection (haibochen via rkanter) 2016-10-21 17:46:17 -07:00
Naganarasimha
7d21c280a8 MAPREDUCE-6777. Typos in 4 log messages. Contributed by Mehran Hassani 2016-09-17 10:19:59 +05:30
Varun Saxena
723facfa40 MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena) 2016-08-20 03:58:16 +05:30
Lei Xu
521f34317a HADOOP-12928. Update netty to 3.10.5.Final to sync with zookeeper. (lei) 2016-07-20 20:38:20 -07:00
Jian He
d8107fcd1c MAPREDUCE-6197. Cache MapOutputLocations in ShuffleHandler. Contributed by Junping Du 2016-06-21 14:25:58 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Eric Payne
cd35b692de MAPREDUCE-6678. Allow ShuffleHandler readahead without drop-behind. Contributed by Nathan Roberts. 2016-05-10 15:45:01 +00:00
Varun Vasudev
2440671a11 MAPREDUCE-6634. Log uncaught exceptions/errors in various thread pools in mapreduce. Contributed by Sidharta Seethana. 2016-02-18 14:18:36 +05:30
Tsuyoshi Ozawa
668e897538 MAPREDUCE-5763. Warn message about httpshuffle in NM logs. Contributed by Akira AJISAKA. 2015-11-09 15:25:31 +09:00
Jason Lowe
ee9b80acf8 MAPREDUCE-6518. Set SO_KEEPALIVE on shuffle connections. Contributed by Chang Li 2015-10-21 21:44:43 +00:00
Jason Lowe
8e615588d5 MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file descriptors. Contributed by Kuhu Shukla 2015-09-10 16:00:17 +00:00
Colin Patrick Mccabe
21d10ccc6e HADOOP-7824. NativeIO.java flags and identifiers must be set correctly for each platform, not hardcoded to their Linux values (Martin Walsh via Colin P. McCabe) 2015-07-31 15:01:34 -07:00
Devaraj K
fffb15bb43 MAPREDUCE-6426. TestShuffleHandler#testGetMapOutputInfo is failing. 
Contributed by zhihai xu.
 2015-07-09 15:00:26 +05:30
Devaraj K
bff67dfe2f MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to 
getMapOutputInfo if mapId is not in the cache. Contributed by zhihai xu.
 2015-07-06 13:46:37 +05:30
Jason Lowe
b3ffa87003 MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections. Contributed by Chang Li 2015-06-05 22:38:31 +00:00
Akira Ajisaka
d335071398 MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 2015-05-07 19:39:24 +09:00
Jian He
808cba3821 YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 2014-12-19 16:56:30 -08:00
Zhijie Shen
1d6e178144 YARN-2347. Consolidated RMStateVersion and NMDBSchemaVersion into Version in yarn-server-common. Contributed by Junping Du. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614838 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 09:27:43 +00:00
Jason Darrell Lowe
60b1e835e0 MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612652 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 19:22:24 +00:00
Karthik Kambatla
f3c3d9e0c6 MAPREDUCE-5652. NM Recovery. ShuffleHandler should handle NM restarts. (Jason Lowe via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1594329 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-13 19:10:48 +00:00
Chris Nauroth
a90a5c2452 MAPREDUCE-5791. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently. Contributed by Nikola Vujic. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580994 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-24 18:57:52 +00:00
Vinod Kumar Vavilapalli
a5c08eed16 MAPREDUCE-5787. Added the ability to keep alive shuffle connections in the MapReduce shuffle-handler. Contributed by Rajesh Balamohan. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580062 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-21 21:43:55 +00:00
Vinod Kumar Vavilapalli
ae29d9ee04 MAPREDUCE-5761. Added a simple log message to denote when encrypted shuffle is on in the shuffle-handler. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1571514 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-25 01:24:26 +00:00
Colin McCabe
efea68dc35 HDFS-5511. improve CacheManipulator interface to allow better unit testing (cmccabe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543676 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-20 00:48:05 +00:00
Sanford Ryza
86dc50c1eb MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when fetch fails (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1537855 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-01 08:51:54 +00:00
Jason Darrell Lowe
e0c99b80d4 MAPREDUCE-5596. Allow configuring the number of threads used to serve shuffle connections. Contributed by Sandy Ryza 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1536711 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-29 13:41:50 +00:00
Siddharth Seth
10a4289ebf YARN-1229. Define constraints on Auxiliary Service names. Change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1526065 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-25 00:35:03 +00:00
Colin McCabe
c1314eb2a3 HDFS-4817. Make HDFS advisory caching configurable on a per-file basis. (Colin Patrick McCabe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1505753 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-22 18:15:18 +00:00
Arun Murthy
11bcd2ed12 MAPREDUCE-5326. Added version to shuffle header. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1496741 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-26 05:49:31 +00:00
Vinod Kumar Vavilapalli
f4d80e91ae YARN-841. Move Auxiliary service to yarn-api, annotate and document it. Contributed by Vinod Kumar Vavilapalli. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1494031 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-18 06:20:37 +00:00
Vinod Kumar Vavilapalli
b9efe6bd4a HADOOP-9649. Promoted YARN service life-cycle libraries into Hadoop Common for usage across all Hadoop projects. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1493650 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-17 06:39:33 +00:00
Vinod Kumar Vavilapalli
0928502029 YARN-530. Defined Service model strictly, implemented AbstractService for robust subclassing and migrated yarn-common services. Contributed by Steve Loughran. 
YARN-117. Migrated rest of YARN to the new service model. Contributed by Steve Louhran.
MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle. Contributed by Steve Loughran.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1492718 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-13 15:54:38 +00:00
Vinod Kumar Vavilapalli
d33534c4fb YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. Contributed by Jian He. 
MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public after YARN-748. Contributed by Jian He.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1489257 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-04 00:34:18 +00:00
Thomas White
9a3379f6b1 MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1489060 13f79535-47bb-0310-9956-ffa450edef68
 2013-06-03 16:53:13 +00:00
Vinod Kumar Vavilapalli
982753dc8e YARN-716. Making ApplicationID immutable. Contributed by Siddharth Seth. 
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after YARN-716. Contributed by Siddharth Seth.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1487994 13f79535-47bb-0310-9956-ffa450edef68
 2013-05-30 20:18:39 +00:00
Vinod Kumar Vavilapalli
47d1ca402f MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading local files. Contributed by Omkar Vinit Joshi. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1481657 13f79535-47bb-0310-9956-ffa450edef68
 2013-05-12 21:59:16 +00:00
Thomas White
9b97df9abe MAPREDUCE-5036. Default shuffle handler port should not be 8080. Contributed by Sandy Ryza. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1478422 13f79535-47bb-0310-9956-ffa450edef68
 2013-05-02 15:58:53 +00:00