big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,444 Commits 2 Branches 2 Tags 264 MiB
137e20cc9b
Commit Graph

210 Commits

Author SHA1 Message Date
Weiwei Yang
9289641020 YARN-7617. Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are started. Contributed by Weiwei Yang. 2017-12-18 10:07:16 +08:00
Weiwei Yang
40b0045ebe YARN-7610. Extend Distributed Shell to support launching job with opportunistic containers. Contributed by Weiwei Yang. 2017-12-06 17:52:41 +08:00
Akira Ajisaka
785f1b0d11
YARN-7407. Moving logging APIs over to slf4j in hadoop-yarn-applications. Contributed by Yeliang Cang. 2017-10-31 16:36:02 +09:00
Yufei Gu
6b7c87c945 YARN-7355. TestDistributedShell should be scheduler agnostic. (Contributed by Haibo) 2017-10-20 11:15:20 -07:00
Wangda Tan
e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Sunil G
7805deed48 YARN-5588. [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Arun Suresh
c41118a7f8 YARN-6978. Add updateContainer API to NMClient. (Kartheek Muthyala via asuresh) 2017-09-07 10:23:12 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Sangjin Lee
092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Daniel Templeton
d81372dfad YARN-6646. Modifier 'static' is redundant for inner enums 
(Contributed by ZhangBing Lin via Daniel Templeton)
 2017-05-26 12:05:48 -07:00
Andrew Wang
539ef5aa2e HFDS-11596. hadoop-hdfs-client jar is in the wrong directory in release tarball. Contributed by Yuanbo Liu. 2017-04-05 16:04:09 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Sangjin Lee
4fa1afdb88 YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 2017-02-16 11:41:04 -08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Daniel Templeton
71a4acf74b YARN-5991. Yarn Distributed Shell does not print throwable t to App Master When failed to start container 
(Contributed by Jim Frankola via Daniel Templeton)
 2017-01-06 14:26:08 -08:00
Akira Ajisaka
209e805430 HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki. 2016-11-28 14:34:57 +09:00
Akira Ajisaka
d3bb69a667 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 2016-10-26 15:32:07 +09:00
Arun Suresh
d6d9cff21b YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 2016-08-30 15:52:29 -07:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Varun Saxena
a3ac1c1943 YARN-5354. TestDistributedShell.checkTimelineV2 fails randomly for concurrent tests. (Sangjin Lee via Varun Saxena). 2016-07-12 20:41:52 +05:30
Sangjin Lee
6cf6ab7b78 Made a number of miscellaneous fixes for javac, javadoc, and checstyle warnings. 2016-07-10 08:46:05 -07:00
Varun Saxena
1ff6833bba YARN-5243. fix several rebase and other miscellaneous issues before merge. (Sangjin Lee via Varun Saxena) 2016-07-10 08:46:03 -07:00
Vrushali Channapattan
3832795e3c YARN-5210 NPE in Distributed Shell while publishing DS_CONTAINER_START event (Varun Saxena via Vrushali C) 2016-07-10 08:46:02 -07:00
Li Lu
0a9b085f05 YARN-5189. Make HBaseTimeline[Reader|Writer]Impl default and move FileSystemTimeline*Impl. (Joep Rottinghuis and Sangjin Lee via gtcarrera9) 2016-07-10 08:46:01 -07:00
Sangjin Lee
d5dfee24b1 YARN-5050. Code cleanup for TestDistributedShell (Li Lu via sjlee) 2016-07-10 08:45:59 -07:00
Varun Saxena
a3cf40e532 YARN-3461. Consolidate flow name/version/run defaults. (Sangjin Lee via Varun Saxena) 2016-07-10 08:45:55 -07:00
Sangjin Lee
0d02ab8729 YARN-3367. Replace starting a separate thread for post entity with event loop in TimelineClient (Naganarasimha G R via sjlee) 2016-07-10 08:45:53 -07:00
Varun Saxena
eb0ac8efb1 YARN-4350. TestDistributedShell fails for V2 scenarios. (Naganarasimha G R via Varun Saxena) 2016-07-10 08:45:50 -07:00
Li Lu
8ef546c1ee YARN-4445. Unify the term flowId and flowName in timeline v2 codebase. 
Contributed by Zhan Zhang.
 2016-07-10 08:45:49 -07:00
Li Lu
89e5c44f9e YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has no 
impact when it's turned off. Contributed by Sangjin Lee.
 2016-07-10 08:45:48 -07:00
Junping Du
477a30f536 YARN-3045. Implement NM writing container lifecycle events to Timeline Service v2. Contributed by Naganarasimha G R. 2016-07-10 08:45:42 -07:00
Sangjin Lee
22e7ae5771 YARN-3792. Test case failures in TestDistributedShell and some issue fixes related to ATSV2 (Naganarasimha G R via sjlee) 
(cherry picked from commit 84f37f1c7eefec6d139cbf091c50d6c06f734323)
 2016-07-10 08:45:38 -07:00
Zhijie Shen
f3c661e8dd YARN-3044. Made RM write app, attempt and optional container lifecycle events to timeline service v2. Contributed by Naganarasimha G R. 2016-07-10 08:45:37 -07:00
Junping Du
47f35a30bb YARN-3391. Clearly define flow ID/ flow run / flow version in API and storage. Contributed by Zhijie Shen 
(cherry picked from commit 68c6232f8423e55b4d152ef3d1d66aeb2d6a555e)
 2016-07-10 08:45:33 -07:00
Zhijie Shen
5712b8f9fd YARN-3334. NM uses timeline client to publish container metrics to new timeline service. Contributed by Junping Du. 2016-07-10 08:45:33 -07:00
Junping Du
d67c9bdb4d YARN-3040. Make putEntities operation be aware of the app's context. Contributed by Zhijie Shen 2016-07-10 08:45:32 -07:00
Junping Du
2188a07e5b YARN-3333. Rename TimelineAggregator etc. to TimelineCollector. Contributed by Sangjin Lee 2016-07-10 08:45:31 -07:00
Zhijie Shen
9b56364080 YARN-3039. Implemented the app-level timeline aggregator discovery service. Contributed by Junping Du. 2016-07-10 08:45:31 -07:00
Zhijie Shen
f0e752c14b YARN-3264. Created backing storage write interface and a POC only FS based storage implementation. Contributed by Vrushali C. 
(cherry picked from commit 821b68d05d246fd57d7b7286eb2ccc075ed1eae8)
 2016-07-10 08:45:30 -07:00
Zhijie Shen
bf54d32750 YARN-3210. Refactored timeline aggregator according to new code organization proposed in YARN-3166. Contributed by Li Lu. 
(cherry picked from commit d3ff7f06cbc66d3a23c2551e7d4c752689f46afe)
 2016-07-10 08:45:30 -07:00
Zhijie Shen
d45ff878c4 YARN-3125. Made the distributed shell use timeline service next gen and add an integration test for it. Contributed by Junping Du and Li Lu. 
(cherry picked from commit bf08f7f0ed4900ce52f98137297dd1a47ba2a536)
 2016-07-10 08:45:30 -07:00
Junping Du
9bdb5bebea Fix OOM issue in DistributedShell. Contributed by Jian He. 2016-07-09 05:09:53 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Junping Du
48c931331c YARN-5131. Distributed shell AM fails when extra container arrives during finishing. Contributed by Wangda Tan. 2016-05-25 06:18:11 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Varun Vasudev
0f25a1bb52 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong. 2016-04-29 16:09:07 +05:30
Xuan
e6c0742012 YARN-4965. Distributed shell AM failed due to ClientHandlerException thrown by jersey. Contributed by Junping Du 2016-04-16 19:39:18 -07:00
Robert Kanter
8286270466 YARN-4436. DistShell ApplicationMaster.ExecBatScripStringtPath is misspelled (mattlamantia via rkanter) 2016-03-29 17:21:29 -07:00
Junping Du
f291d82cd4 YARN-4545. Allow YARN distributed shell to use ATS v1.5 APIs. Li Lu via junping_du 2016-03-14 08:28:38 -07:00
Steve Loughran
e9a622606f YARN-4629. Distributed shell breaks under strong security. (Daniel Templeton via stevel) 2016-02-10 13:01:10 +00:00
Jian He
f385851141 YARN-4596. SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities. Contributed by Li Lu 2016-01-18 16:58:39 -08:00
Wangda Tan
7ff280fca9 YARN-1509. Make AMRMClient support send increase container request and get increased/decreased containers. (Meng Ding via wangda) 2015-11-12 11:50:00 -08:00
Wangda Tan
c99925d6dd YARN-1510. Make NMClient support change container resources. (Meng Ding via wangda) 2015-11-10 11:45:46 -08:00
Wangda Tan
0783184f4b YARN-4326. Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188. (Meng Ding via wangda) 2015-11-03 11:18:34 -08:00
Steve Loughran
446212a39e YARN-4296. DistributedShell Log.info is not friendly. (Xiaowei Wang via stevel) 2015-10-24 12:55:36 +01:00
Steve Loughran
14f8dd0423 YARN-4289. TestDistributedShell failing with bind exception. (Brahma Reddy Battula via stevel) 2015-10-24 12:48:15 +01:00
Jason Lowe
2b2bd92146 YARN-3950. Add unique SHELL_ID environment variable to DistributedShell. Contributed by Robert Kanter 2015-07-29 15:16:40 +00:00
Jian He
7438966586 YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted. Contributed by Varun Vasudev 2015-05-19 14:20:31 -07:00
Devaraj K
241a72af0d YARN-2784. Make POM project names consistent. Contributed by Rohith. 2015-05-08 19:23:53 +05:30
Akira Ajisaka
918af8efff YARN-3577. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 2015-05-07 13:33:03 +09:00
Akira Ajisaka
5ce3a77f3c YARN-3444. Fix typo capabililty. Contributed by Gabor Liptak. 2015-04-25 06:08:16 +09:00
Vinod Kumar Vavilapalli
f5fe35e297 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv) 2015-04-23 11:19:55 -07:00
Jian He
aa4a192feb YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the correct user. Contributed by Zhijie Shen 2015-04-23 11:07:26 -07:00
Hitesh Shah
265ed1fe80 YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 2015-04-08 14:13:10 -07:00
Jonathan Eagles
8180e676ab YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 2015-03-13 12:04:30 -05:00
Tsuyoshi Ozawa
24aa462673 YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. Contributed by Wangda Tan. 2015-01-23 20:37:05 +09:00
Junping Du
d468c9aaf1 YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 2014-12-23 20:04:33 -08:00
Junping Du
856b46cf47 Revert ca3381de61 and 20674255ad as missing JIRA number in commit message 2014-12-23 20:02:33 -08:00
Junping Du
ca3381de61 Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 2014-12-23 18:57:16 -08:00
Jian He
a4f2995b9e Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-12-15 11:04:09 -08:00
Zhijie Shen
51af8d367d YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 2014-12-01 19:04:04 -08:00
Jonathan Eagles
c298a9a845 YARN-2375. Allow enabling/disabling timeline server per framework. (Mit Desai via jeagles) 2014-11-20 23:34:35 -06:00
Vinod Kumar Vavilapalli
737d9284c1 YARN-2607. Fixed issues in TestDistributedShell. Contributed by Wangda Tan. 2014-11-08 11:00:57 -08:00
Vinod Kumar Vavilapalli
2ac1be7dec YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. 2014-11-07 10:39:37 -08:00
Zhijie Shen
a8c1202220 YARN-2769. Fixed the problem that timeline domain is not set in distributed shell AM when using shell_command on Windows. Contributed by Varun Vasudev. 2014-10-29 12:12:30 -07:00
Jian He
f6b963fdfc YARN-2502. Changed DistributedShell to support node labels. Contributed by Wangda Tan 2014-10-27 20:13:00 -07:00
Zhijie Shen
1d4612f5ad YARN-2629. Made the distributed shell use the domain-based timeline ACLs. Contributed by Zhijie Shen. 2014-10-09 12:59:47 -07:00
Zhijie Shen
52bbe0f11b YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 2014-10-01 15:38:11 -07:00
XuanGong
8e5d6713cf YARN-2557. Add a parameter "attempt_Failures_Validity_Interval" into 
DistributedShell. Contributed by Xuan Gong
 2014-09-16 10:58:18 -07:00
Hitesh Shah
3de66011c2 YARN-2450. Fix typos in log messages. Contributed by Ray Chiang. 2014-08-29 11:16:36 -07:00
Zhijie Shen
1a535f8dcd YARN-2070. Made DistributedShell publish the short user name to the timeline server. Contributed by Robert Kanter. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617837 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-13 21:07:18 +00:00
Jian He
9ebeac16eb YARN-2374. Fixed TestDistributedShell#testDSShell failure due to hostname dismatch. Contributed by Varun Vasudev 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616302 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-06 18:58:50 +00:00
Jian He
7e54b1c6d9 YARN-2354. DistributedShell may allocate more containers than client specified after AM restarts. Contributed by Li Lu 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614538 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-30 03:58:59 +00:00
Jian He
f6c723ff0c YARN-2295. Refactored DistributedShell to use public APIs of protocol records. Contributed by Li Lu 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612626 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 17:49:14 +00:00
Zhijie Shen
28058894dd YARN-2081. Fixed TestDistributedShell failure after YARN-1962. Contributed by Zhiguo Hong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1596724 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-22 01:16:14 +00:00
Haohui Mai
d777cb8996 HADOOP-10563. Remove the dependency of jsp in trunk. Contributed by Haohui Mai. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1592069 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-02 22:26:02 +00:00
Vinod Kumar Vavilapalli
d6a6e982a8 YARN-1926. Changed DistributedShell to use appIDs as unique identifiers for HDFS paths and thus avoid test failures on Windows. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1586562 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-11 04:06:40 +00:00
Vinod Kumar Vavilapalli
6c2a0ce30b YARN-1908. Fixed DistributedShell to not fail in secure clusters. Contributed by Vinod Kumar Vavilapalli and Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1585849 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-08 22:11:52 +00:00
Zhijie Shen
3af3b0bda9 YARN-1873. Fixed TestDistributedShell failure when the test cases are out of order. Contributed by Mit Desai. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1582060 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-26 21:18:38 +00:00
Zhijie Shen
0f1eda6bbf YARN-1838. Enhanced timeline service getEntities API to get entities from a given entity ID or insertion timestamp. Contributed by Billie Rinaldi. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580960 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-24 18:14:13 +00:00
Vinod Kumar Vavilapalli
bf428cc474 YARN-1855. Made Application-history server to be optional in MiniYARNCluster and thus avoid the failure of TestRMFailover#testRMWebAppRedirect. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1579838 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-21 00:18:33 +00:00
Zhijie Shen
fd1c424548 YARN-1690. Made DistributedShell send timeline entities+events. Contributed by Mayank Bansal. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1579123 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-19 03:49:29 +00:00
Jonathan Turner Eagles
7944eab64a YARN-1136. Replace junit.framework.Assert with org.junit.Assert (Chen He via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578539 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 20:03:35 +00:00
Vinod Kumar Vavilapalli
96e0ca2d27 YARN-1824. Improved NodeManager and clients to be able to handle cross platform application submissions. Contributed by Jian He. 
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to handle cross platform application submissions. Contributed by Jian He.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578135 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-16 18:32:05 +00:00