big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,494 Commits 2 Branches 2 Tags 264 MiB
f49843a988
Commit Graph

1472 Commits

Author SHA1 Message Date
Karthik Kambatla
652679aa8a YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 2017-02-14 13:39:41 -08:00
Arun Suresh
aaf106fde3 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 2017-02-14 06:09:10 -08:00
Karthik Kambatla
646c6d6509 YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 2017-02-13 11:26:30 -08:00
Karthik Kambatla
08f93978f3 YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 2017-02-09 14:45:04 -08:00
Karthik Kambatla
9b85053583 YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:09:32 -08:00
Wangda Tan
5fb723bb77 YARN-5889. Improve and refactor user-limit calculation in Capacity Scheduler. (Sunil G via wangda) 2017-02-09 10:23:50 -08:00
Daniel Templeton
87852b6ef4 YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 2017-01-30 21:44:48 -08:00
Sunil G
165f07f51a YARN-6123. [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed. Contributed by Wangda Tan. 2017-01-27 18:37:51 +05:30
Daniel Templeton
f85b74ccf9 YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-01-26 10:31:09 -08:00
Karthik Kambatla
abedb8a9d8 YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 2017-01-25 12:17:28 -08:00
Rohith Sharma K S
0101267d9d YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 2017-01-24 10:59:55 +05:30
Wangda Tan
ce832059db YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-23 10:52:14 -08:00
Jason Lowe
69fa81679f YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 2017-01-23 17:12:51 +00:00
Sunil G
3fa0d540df YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 2017-01-23 12:19:35 +05:30
Wangda Tan
72054a817d YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 2017-01-18 13:31:33 -08:00
Karthik Kambatla
e224c96234 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 2017-01-17 17:01:31 -08:00
Sunil G
d3170f9eba YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 2017-01-13 18:22:29 +05:30
Naganarasimha
a6b06f7179 YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 2017-01-12 06:40:41 +05:30
Daniel Templeton
7979939428 YARN-5554. MoveApplicationAcrossQueues does not check user permission on the target queue 
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
 2017-01-11 14:34:10 -08:00
Sunil G
9594c35dcb YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 2017-01-10 14:08:01 +05:30
Junping Du
945db55f2e YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 2017-01-09 18:14:46 -08:00
Wangda Tan
287d3d6804 YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 2017-01-09 08:40:39 -08:00
Daniel Templeton
ac16400e1f YARN-6022. Revert changes of AbstractResourceRequest (Contributed by Wangda Tan via Daniel Templeton) 2017-01-07 08:24:30 -08:00
Naganarasimha
a59df15757 YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-07 08:11:58 +05:30
Jian He
020316458d YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 2017-01-06 18:19:09 -08:00
Arun Suresh
2977bc6a14 YARN-6040. Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes. (Wangda Tan via asuresh) 2017-01-06 09:59:57 -08:00
Wangda Tan
0a55bd841e YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 2017-01-05 10:31:05 -08:00
Rohith Sharma K S
e49e0a6e37 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S. 2017-01-04 12:09:29 +05:30
Wangda Tan
f69a107aec YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
 2017-01-03 14:53:36 -08:00
Wangda Tan
ab1faa4ba8 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" to add correct JIRA number 
This reverts commit e0f2379312.
 2017-01-03 14:53:13 -08:00
Wangda Tan
e0f2379312 Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 2017-01-03 14:46:41 -08:00
Daniel Templeton
165d01a73e YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton) 2016-12-30 08:41:52 -08:00
Robert Kanter
f216276d21 YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter) 2016-12-28 15:21:52 -08:00
Sunil G
9ca54f4810 YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2016-12-28 22:48:19 +05:30
Wangda Tan
0840b4329b YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda) 2016-12-27 21:18:33 -08:00
Robert Kanter
1bbd023275 YARN-5962. Spelling errors in logging and exceptions for resource manager code (gsohn via rkanter) 2016-12-27 13:35:34 -08:00
Daniel Templeton
e216e8e233 HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-27 13:03:16 -08:00
Arun Suresh
ac1e5d4f77 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 2016-12-27 12:40:44 -08:00
Karthik Kambatla
c3973e7080 YARN-5969. FairShareComparator: Cache value of getResourceUsage for better performance. (zhangshilong via kasha) 2016-12-27 12:30:35 -08:00
Karthik Kambatla
4d3f73acc0 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 2016-12-23 11:40:56 -08:00
Sunil G
ef2dd7b78c YARN-5650. Render Application Timeout value in web UI. Contributed by Akhil PB. 2016-12-19 11:44:03 +05:30
Arun Suresh
2273a74c1f YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 2016-12-16 08:14:34 -08:00
Jian He
ada876cd1d Revert YARN-4126. RM should not issue delegation tokens in unsecure mode. 2016-12-13 20:49:54 -08:00
Robert Kanter
4b149a1e77 YARN-4457. Cleanup unchecked types for EventHandler (templedf via rkanter) 2016-12-09 17:28:25 -08:00
Jian He
a6410a542e YARN-5709. Cleanup leader election configs and pluggability. Contribtued by Karthik Kambatla 2016-12-09 17:00:37 -08:00
Arun Suresh
b0aace21b1 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 2016-12-09 16:41:25 -08:00
Sunil G
74d0066d33 YARN-5970. Validate application update timeout request parameters. Contributed by Rohith Sharma K S. 2016-12-08 15:53:56 +05:30
Daniel Templeton
9f5d2c4fff YARN-5136. Error in handling event type APP_ATTEMPT_REMOVED to the scheduler 
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
 2016-12-07 11:12:14 -08:00
Sunil
ab923a53fc YARN-5965. Retrospect ApplicationReport#getApplicationTimeouts. Contributed by Rohith Sharma K S 2016-12-08 00:27:25 +05:30
Rohith Sharma K S
563480dccd YARN-5932. Retrospect moveApplicationToQueue in align with YARN-5611. Contributed by Sunil G. 2016-12-07 10:39:14 +05:30
Naganarasimha
f3b8ff54ab YARN-5921. Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState. Contributed by Varun Saxena 2016-12-06 06:53:38 +05:30
Jian He
f885160f4a YARN-5746. The state of the parentQueue and its childQueues should be synchronized. Contributed by Xuan Gong 2016-12-02 16:17:31 -08:00
Daniel Templeton
5bd18c49bd YARN-5929. Missing scheduling policy in the FS queue metric. (Contributed by Yufei Gu via Daniel Templeton) 2016-12-02 13:55:42 -08:00
Li Lu
69fb70c31a YARN-5761. Separate QueueManager from Scheduler. (Xuan Gong via gtcarrera9) 2016-11-30 13:38:42 -08:00
Daniel Templeton
b3befc021b YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 2016-11-30 09:50:33 -08:00
Daniel Templeton
3b9d3acd20 YARN-5890. FairScheduler should log information about AM-resource-usage and max-AM-share for queues 
(Contributed by Yufei Gu via Daniel Templeton)
 2016-11-29 12:46:05 -08:00
Daniel Templeton
25f9872be6 YARN-5774. MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
if set yarn.scheduler.minimum-allocation-mb to 0. (Contributed by Yufei Gu via Daniel Templeton)
 2016-11-29 09:40:49 -08:00
Sunil
eb0a483ed0 YARN-4206. Add Application timeouts in Application report and CLI. Contributed by Rohith Sharma K S. 2016-11-24 18:18:42 +05:30
Daniel Templeton
10468529a9 YARN-4752. Improved preemption in FairScheduler. (kasha) 
Contains:
YARN-5605. Preempt containers (all on one node) to meet the requirement of starved applications
YARN-5821. Drop left-over preemption-related code and clean up method visibilities in the Schedulable hierarchy
YARN-5783. Verify identification of starved applications.
YARN-5819. Verify fairshare and minshare preemption
YARN-5885. Cleanup YARN-4752 branch for merge

Change-Id: Iee0962377d019dd64dc69a020725d2eaf360858c
 2016-11-23 19:48:59 -10:00
Jian He
1f12867a69 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-23 16:25:39 -08:00
Arun Suresh
005850b28f YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh) 2016-11-23 09:53:31 -08:00
Daniel Templeton
83cc7263af YARN-5722. FairScheduler hides group resolution exceptions when assigning queue (Contributed by Wilfred Spiegelenburg via Daniel Templeton) 2016-11-22 11:16:00 -08:00
Rohith Sharma K S
a926f895c1 YARN-5865. Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611. Contributed by Sunil G. 2016-11-22 14:49:15 +05:30
Rohith Sharma K S
d65603517e YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee. 2016-11-16 15:14:00 +05:30
Arun Suresh
3219b7b4ac YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 2016-11-15 07:56:25 -08:00
Rohith Sharma K S
b7070f3308 YARN-5874. RM -format-state-store and -remove-application-from-state-store commands fail with NPE. Contributed by Varun Saxena. 2016-11-15 10:58:25 +05:30
Rohith Sharma K S
296c5de0cf YARN-5873. RM crashes with NPE if generic application history is enabled. Contributed by Varun Saxena. 2016-11-15 10:56:28 +05:30
Karthik Kambatla
79448d4ab1 YARN-4033. In FairScheduler, parent queues should also display queue status. (Siqi Li via kasha) 2016-11-13 21:38:36 -08:00
Jian He
fad9609d13 YARN-5825. ProportionalPreemptionalPolicy should use readLock over LeafQueue instead of synchronized block. Contributed by Sunil G 2016-11-11 15:16:21 -08:00
Wangda Tan
ede1a473f5 YARN-5552. Add Builder methods for common yarn API records. (Tao Jie via wangda) 2016-11-11 13:34:56 -08:00
Naganarasimha
503e73e849 YARN-5545. Fix issues related to Max App in capacity scheduler. Contributed by Bibin A Chundatt 2016-11-11 20:48:31 +05:30
Eric Payne
93eeb13164 YARN-4218. Metric for resource*time that was preempted. Contributed by Chang Li. 2016-11-10 22:35:12 +00:00
Karthik Kambatla
86ac1ad9fd YARN-5453. FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource. (sandflee via kasha) 2016-11-09 23:44:02 -08:00
Jian He
bcc15c6290 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:08:05 -08:00
Naganarasimha
edbee9e609 YARN-4498. Application level node labels stats to be available in REST (addendum patch). Contributed by Bibin A Chundatt. 2016-11-10 05:00:05 +05:30
Daniel Templeton
59ee8b7a88 YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted app 
is in ACCEPTED state in Fair Scheduler (Contributed by Yufei Gu)
 2016-11-09 13:11:37 -08:00
Arun Suresh
283fa33feb YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 2016-11-09 00:11:25 -08:00
Jason Lowe
3f93ac0733 YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 2016-11-08 22:01:26 +00:00
Jian He
de3b4aac56 YARN-5716. Add global scheduler interface definition and update CapacityScheduler to use it. Contributed by Wangda Tan 2016-11-07 10:14:39 -08:00
Wangda Tan
c00b5d1e51 YARN-5804. New UI2 is not able to launch with jetty 9 upgrade post HADOOP-10075. (Sunil G via wangda) 
(cherry picked from commit ddebaa6b7ae1031899139d153d57df5387d60542)
 2016-11-06 13:13:31 -08:00
Wangda Tan
3de0da2a76 YARN-5698. [YARN-3368] Launch new YARN UI under hadoop web app port. (Sunil G via wangda) 2016-11-06 13:13:31 -08:00
Wangda Tan
817d5f4c96 YARN-4849. Addendum patch to fix javadocs. (Sunil G via wangda) 2016-11-06 13:13:31 -08:00
Wangda Tan
c85cc3b56e YARN-4515. [YARN-3368] Support hosting web UI framework inside YARN RM. (Sunil G via wangda) 
YARN-5000. [YARN-3368] App attempt page is not loading when timeline server is not started (Sunil G via wangda)
YARN-5038. [YARN-3368] Application and Container pages shows wrong values when RM is stopped. (Sunil G via wangda)
 2016-11-06 13:13:31 -08:00
Jason Lowe
6bb741ff0e YARN-5837. NPE when getting node status of a decommissioned node after an RM restart. Contributed by Robert Kanter 2016-11-04 22:20:21 +00:00
Arun Suresh
0aafc122d4 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 2016-11-04 07:31:54 -07:00
Sunil
19b3779ae7 YARN-5802. updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy. Contributred by Bibin A Chundatt 2016-11-04 16:07:28 +05:30
Jason Lowe
352cbaa7a5 YARN-4862. Handle duplicate completed containers in RMNodeImpl. Contributed by Rohith Sharma K S 2016-11-03 13:54:31 +00:00
Varun Saxena
7d2d8d25ba YARN-5788. Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode (Bibin A Chundatt via Varun Saxena) 2016-11-01 15:32:04 +05:30
Karthik Kambatla
733aa99313 YARN-4998. Minor cleanup to UGI use in AdminService. (Daniel Templeton via kasha) 2016-10-31 16:26:01 -07:00
Wangda Tan
90dd3a8148 YARN-2009. CapacityScheduler: Add intra-queue preemption for app priority support. (Sunil G via wangda) 2016-10-31 15:18:31 -07:00
Daniel Templeton
2528bea67f YARN-4396. Log the trace information on FSAppAttempt#assignContainer (Contributed by Yiqun Li via Daniel Templeton) 2016-10-31 13:34:25 -07:00
Naganarasimha
e0bebbbcdd YARN-4498. Application level node labels stats to be available in REST. Contributed by Bibin A Chundatt 2016-10-31 04:38:20 +05:30
Arun Suresh
aa3cab1eb2 YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 2016-10-29 02:03:57 -07:00
Varun Saxena
1c8ab41e8b YARN-5773. RM recovery too slow due to LeafQueue#activateApplications (Bibin A Chundatt via Varun Saxena) 2016-10-29 13:47:39 +05:30
Jason Lowe
1eae719bce YARN-4963. capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable. Contributed by Nathan Roberts 2016-10-28 17:30:15 +00:00
Karthik Kambatla
4df8ed63ed YARN-4743. FairSharePolicy breaks TimSort assumption. (Zephyr Guo and Yufei Gu via kasha) 2016-10-27 17:45:48 -07:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Carlo Curino
892a8348fc YARN-3432. Cluster metrics have wrong Total Memory when there is reserved memory on CS. (Brahma Reddy Battula via curino) 2016-10-27 15:15:49 -07:00
Daniel Templeton
b98fc8249f YARN-4710. Reduce logging application reserved debug info in FSAppAttempt#assignContainer (Contributed by Yiqun Lin via Daniel Templeton) 2016-10-27 14:42:19 -07:00
Naganarasimha
b3c15e4ef7 YARN-5420. Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary. Contributed by Sunil G. 2016-10-27 18:22:07 +05:30
Karthik Kambatla
a71fc81655 YARN-5754. Null check missing for earliest in FifoPolicy. (Yufei Gu via kasha) 2016-10-24 22:19:23 -07:00
Karthik Kambatla
754cb4e30f YARN-5047. Refactor nodeUpdate across schedulers. (Ray Chiang via kasha) 2016-10-20 21:17:48 -07:00
Karthik Kambatla
a064865abf YARN-4911. Bad placement policy in FairScheduler causes the RM to crash 2016-10-20 20:57:04 -07:00
Xuan
b733a6f862 YARN-5718. TimelineClient (and other places in YARN) shouldn't over-write HDFS client retry settings which could cause unexpected behavior. Contributed by Junping Du. 2016-10-18 11:06:47 -07:00
Varun Saxena
b154d3edce YARN-5743. [Atsv2] Publish queue name and RMAppMetrics to ATS (Rohith Sharma K S via Varun Saxena) 2016-10-18 23:32:52 +05:30
Sangjin Lee
1f304b0c7f YARN-5699. Retrospect yarn entity fields which are publishing in events info fields. Contributed by Rohith Sharma K S. 2016-10-15 13:54:40 -07:00
Karthik Kambatla
6476934ae5 YARN-5677. RM should transition to standby when connection is lost for an extended period. (Daniel Templeton via kasha) 2016-10-11 22:07:10 -07:00
Karthik Kambatla
6378845f9e YARN-4464. Lower the default max applications stored in the RM and store. (Daniel Templeton via kasha) 2016-10-11 21:42:08 -07:00
Rohith Sharma K S
4d2f380d78 YARN-5101. YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order. Contributed by Sunil G. 2016-10-06 20:42:36 +05:30
Jian He
31f8da22d0 YARN-3139. Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler. Contributed by Wangda Tan 2016-10-04 17:23:13 -07:00
Karthik Kambatla
736d33cddd YARN-4767. Network issues can cause persistent RM UI outage. (Daniel Templeton via kasha) 2016-10-03 14:35:57 -07:00
Karthik Kambatla
0da54e8848 YARN-5672. FairScheduler: Wrong queue name in log when adding application. (Wilfred Spiegelenburg via kasha) 2016-10-03 06:03:46 -07:00
Naganarasimha
6e130c308c YARN-4855. Should check if node exists when replace nodelabels. Contributeed by Tao Jie 2016-10-03 02:02:26 -04:00
Karthik Kambatla
82857037b6 YARN-5678. Log demand as demand in FSLeafQueue and FSParentQueue. (Yufei Gu via kasha) 2016-10-02 22:09:43 -07:00
Subru Krishnan
3a3697deab YARN-5384. Expose priority in ReservationSystem submission APIs. (Sean Po via Subru). 2016-09-30 19:41:43 -07:00
Arun Suresh
10be45986c YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 2016-09-29 15:11:41 -07:00
Jian He
2ae5a3a5bf YARN-4205. Add a service for monitoring application life time out. Contributed by Rohith Sharma K S 2016-09-29 22:00:31 +08:00
Robert Kanter
bcb2528a51 YARN-5400. Light cleanup in ZKRMStateStore (templedf via rkanter) 2016-09-28 14:56:41 -07:00
Varun Saxena
9b0fd01d2e YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun Saxena) 2016-09-28 16:10:10 +05:30
Wangda Tan
1831be8e73 YARN-3142. Improve locks in AppSchedulingInfo. (Varun Saxena via wangda) 2016-09-27 11:54:55 -07:00
Naganarasimha
ebf528c443 YARN-5660. Wrong audit constants are used in Get/Put of priority in RMWebService. Contributed by Rohith Sharma K S 2016-09-27 16:37:59 +05:30
Akira Ajisaka
14a696f369 YARN-5663. Small refactor in ZKRMStateStore. Contributed by Oleksii Dymytrov. 2016-09-26 15:00:01 +09:00
Naganarasimha
d0372dc613 YARN-3692. Allow REST API to set a user generated message when killing an application. Contributed by Rohith Sharma K S 2016-09-23 06:30:49 +05:30
Arun Suresh
9f03b403ec YARN-5656. Fix ReservationACLsTestBase. (Sean Po via asuresh) 2016-09-20 12:27:17 -07:00
Wangda Tan
e45307c9a0 Addendum patch for fix javadocs failure which is caused by YARN-3141. (wangda) 2016-09-20 11:21:01 -07:00
Jian He
2b66d9ec5b YARN-3140. Improve locks in AbstractCSQueue/LeafQueue/ParentQueue. Contributed by Wangda Tan 2016-09-20 15:03:31 +08:00
Jason Lowe
7558dbbb48 YARN-5540. Scheduler spends too much time looking at empty priorities. Contributed by Jason Lowe 2016-09-19 20:31:35 +00:00
Jian He
b8a30f2f17 YARN-3141. Improve locks in SchedulerApplicationAttempt/FSAppAttempt/FiCaSchedulerApp. Contributed by Wangda Tan 2016-09-19 17:08:01 +08:00
Naganarasimha
4174b9756c YARN-5642. Typos in 9 log messages. Contributed by Mehran Hassani 2016-09-17 10:35:39 +05:30
Karthik Kambatla
c0e492e50f YARN-5616. Clean up WeightAdjuster. (Yufei Gu via kasha) 2016-09-06 22:40:20 -07:00
Karthik Kambatla
f6ea9be547 YARN-5264. Store all queue-specific information in FSQueue. (Yufei Gu via kasha) 2016-09-02 14:56:29 -07:00
Ray Chiang
378f624a39 YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang) 2016-09-02 11:14:35 -07:00
Varun Vasudev
05f5c0f631 YARN-5555. Scheduler UI: "% of Queue" is inaccurate if leaf queue is hierarchically nested. Contributed by Eric Payne. 2016-09-02 16:02:01 +05:30
Karthik Kambatla
74f4bae455 YARN-5566. Client-side NM graceful decom is not triggered when jobs finish. (Robert Kanter via kasha) 2016-09-01 14:44:01 -07:00
Arun Suresh
d6d9cff21b YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 2016-08-30 15:52:29 -07:00
Subru Krishnan
b930dc3ec0 YARN-5327. API changes required to support recurring reservations in the YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru). 2016-08-26 16:58:47 -07:00
Naganarasimha
46e02ab719 YARN-3940. Application moveToQueue should check NodeLabel permission. Contributed by Bibin A Chundatt 2016-08-26 20:19:11 +05:30
Naganarasimha
27c3b86252 YARN-5564. Fix typo in RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE. Contributed by Ray Chiang 2016-08-26 08:47:21 +05:30
Wangda Tan
444b2ea7af YARN-3388. Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit. (Nathan Roberts via wangda) 2016-08-19 16:28:32 -07:00
Varun Saxena
091dd19e86 YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena) 2016-08-19 17:30:17 +05:30
Varun Saxena
8aed374182 Revert "YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena)" 
This reverts commit 59557e85a4.
 2016-08-19 16:14:16 +05:30
Varun Saxena
59557e85a4 YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena) 2016-08-19 15:01:48 +05:30
Junping Du
0da69c324d YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 
(cherry picked from commit d464483bf7f0b3e3be3ba32cd6c3eee546747ab5)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
 2016-08-18 07:27:23 -07:00
Karthik Kambatla
20f0eb871c YARN-4702. FairScheduler: Allow setting maxResources for ad hoc queues. (Daniel Templeton via kasha) 2016-08-17 17:40:20 -07:00
Varun Saxena
23c6e3c4e4 YARN-5476. Non existent application reported as ACCEPTED by YarnClientImpl (Junping Du via Varun Saxena) 2016-08-12 20:37:58 +05:30
Naganarasimha
874577a67d YARN-4833. For Queue AccessControlException client retries multiple times on both RM. Contributed by Bibin A Chundatt 2016-08-12 01:09:41 +05:30
Naganarasimha
89c0bff719 YARN-5495. Remove import wildcard in CapacityScheduler. Contributed by Ray Chiang. 2016-08-11 00:35:19 +05:30
Jason Lowe
5c95bb315b YARN-5382. RM does not audit log kill request for active applications. Contributed by Vrushali C 2016-08-10 18:25:54 +00:00
Jason Lowe
e0b570dffb YARN-5483. Optimize RMAppAttempt#pullJustFinishedContainers. Contributed by sandflee 2016-08-10 18:04:18 +00:00
Arun Suresh
82c9e06101 YARN-5457. Refactor DistributedScheduling framework to pull out common functionality. (asuresh) 2016-08-09 00:42:29 -07:00
Naganarasimha
d81b8163b4 YARN-4624. NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI. Contributed by Brahma Reddy Battula 2016-08-06 01:13:36 +05:30
Wangda Tan
3f100d76ff YARN-4888. Changes in scheduler to identify resource-requests explicitly by allocation-id. (Subru Krishnan via wangda) 2016-08-05 10:43:35 -07:00
Wangda Tan
e0d131f055 YARN-4091. Add REST API to retrieve scheduler activity. (Chen Ge and Sunil G via wangda) 2016-08-05 10:27:34 -07:00
Rohith Sharma K S
d9a354c2f3 YARN-5333. Some recovered apps are put into default queue when RM HA. Contributed by Jun Gong. 2016-08-05 21:35:49 +05:30
Jason Lowe
4d92aefd35 YARN-4280. CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster. Contributed by Kuhu Shukla 2016-08-03 18:53:14 +00:00
Arun Suresh
e5766b1dbe YARN-5113. Refactoring and other clean-up for distributed scheduling. (Konstantinos Karanasos via asuresh) 2016-07-31 11:48:25 -07:00
Subru Krishnan
4e756d7271 YARN-5203.Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API. Contributed by Ellen Hui. 2016-07-28 16:03:24 -07:00
Wangda Tan
d62e121ffc YARN-5195. RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. (sandflee via wangda) 2016-07-26 21:22:59 -07:00
Wangda Tan
49969b16cd YARN-5342. Improve non-exclusive node partition resource allocation in Capacity Scheduler. (Sunil G via wangda) 2016-07-26 18:14:09 -07:00
Arun Suresh
5aace38b74 YARN-5392. Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerRequestKey. (asuresh and subru) 2016-07-26 14:54:03 -07:00
Chris Douglas
d383bfdcd4 YARN-5164. Use plan RLE to improve CapacityOverTimePolicy efficiency 2016-07-25 16:37:50 -07:00
Akira Ajisaka
c63afdbe14 YARN-4883. Make consistent operation name in AdminService. Contributed by Kai Sasaki. 2016-07-20 16:51:01 -07:00
Arun Suresh
cda0a280dd YARN-5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh) 2016-07-19 10:43:37 -07:00
Arun Suresh
5f2d33a551 Revert "YARN=5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh)" 
This reverts commit e905a42a2c.
 2016-07-19 10:43:19 -07:00
Ray Chiang
f5f1c81e7d YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2016-07-15 14:38:50 -07:00
Arun Suresh
e905a42a2c YARN=5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh) 2016-07-15 14:35:12 -07:00
Wangda Tan
24db9167f1 YARN-4484. Available Resource calculation for a queue is not correct when used with labels. (Sunil G via wangda) 2016-07-15 11:40:12 -07:00
Varun Saxena
06c56ff79b YARN-5353. ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena). 2016-07-13 07:55:34 +05:30
Jian He
819224dcf9 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:36:20 -07:00
Sangjin Lee
6cf6ab7b78 Made a number of miscellaneous fixes for javac, javadoc, and checstyle warnings. 2016-07-10 08:46:05 -07:00
Vrushali
6d943038f6 Cleanup changes during rebase with trunk (Vrushali C) 2016-07-10 08:46:04 -07:00
Varun Saxena
1ff6833bba YARN-5243. fix several rebase and other miscellaneous issues before merge. (Sangjin Lee via Varun Saxena) 2016-07-10 08:46:03 -07:00
Sangjin Lee
702236129b YARN-5095. flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled (Varun Saxena via sjlee) 2016-07-10 08:46:00 -07:00
Sangjin Lee
a1b6d7456f YARN-5018. Online aggregation logic should not run immediately after collectors got started (Li Lu via sjlee) 2016-07-10 08:45:59 -07:00
Li Lu
c2055a97d5 YARN-3150. Documenting the timeline service v2. (Sangjin Lee and Vrushali C via gtcarrera9) 2016-07-10 08:45:57 -07:00
Varun Saxena
a3cf40e532 YARN-3461. Consolidate flow name/version/run defaults. (Sangjin Lee via Varun Saxena) 2016-07-10 08:45:55 -07:00
Sangjin Lee
960af7d471 YARN-4409. Fix javadoc and checkstyle issues in timelineservice code (Varun Saxena via sjlee) 2016-07-10 08:45:53 -07:00
Naganarasimha
06f0b50a28 YARN-4644. TestRMRestart fails and findbugs issue in YARN-2928 branch (Varun Saxena via Naganarasimha G R) 2016-07-10 08:45:52 -07:00
Li Lu
34f02f07d5 Rebase to latest trunk 2016-07-10 08:45:51 -07:00
Varun Saxena
829cceebc0 YARN-3586. RM to only get back addresses of Collectors that NM needs to know. 
(Junping Du via Varun Saxena).
 2016-07-10 08:45:50 -07:00
Varun Saxena
c4d7bbda5c YARN-4460. [Bug fix] RM fails to start when SMP is enabled. (Li Lu via Varun Saxena) 2016-07-10 08:45:49 -07:00
Xuan
2e2dbf59d1 YARN-4392. ApplicationCreatedEvent event time resets after RM 
restart/failover. Contributed by Naganarasimha G R and Xuan Gong

(cherry picked from commit 4546c7582b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
 2016-07-10 08:45:49 -07:00
Li Lu
89e5c44f9e YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has no 
impact when it's turned off. Contributed by Sangjin Lee.
 2016-07-10 08:45:48 -07:00
Sangjin Lee
10ec5586fb YARN-4129. Refactor the SystemMetricPublisher in RM to better support newer events (Naganarasimha G R via sjlee) 2016-07-10 08:45:46 -07:00
Sangjin Lee
8d9476ec5f YARN-4058. Miscellaneous issues in NodeManager project (Naganarasimha G R via sjlee) 2016-07-10 08:45:43 -07:00
Sangjin Lee
22e7ae5771 YARN-3792. Test case failures in TestDistributedShell and some issue fixes related to ATSV2 (Naganarasimha G R via sjlee) 
(cherry picked from commit 84f37f1c7eefec6d139cbf091c50d6c06f734323)
 2016-07-10 08:45:38 -07:00
Zhijie Shen
f3c661e8dd YARN-3044. Made RM write app, attempt and optional container lifecycle events to timeline service v2. Contributed by Naganarasimha G R. 2016-07-10 08:45:37 -07:00
Sangjin Lee
dc1f306fdc YARN-3562. unit tests failures and issues found from findbug from earlier ATS checkins (Naganarasimha G R via sjlee) 2016-07-10 08:45:35 -07:00
Sangjin Lee
11e8905d8d YARN-3390. Reuse TimelineCollectorManager for RM (Zhijie Shen via sjlee) 
(cherry picked from commit 58221188811e0f61d842dac89e1f4ad4fd8aa182)
 2016-07-10 08:45:33 -07:00
Junping Du
47f35a30bb YARN-3391. Clearly define flow ID/ flow run / flow version in API and storage. Contributed by Zhijie Shen 
(cherry picked from commit 68c6232f8423e55b4d152ef3d1d66aeb2d6a555e)
 2016-07-10 08:45:33 -07:00
Zhijie Shen
5712b8f9fd YARN-3334. NM uses timeline client to publish container metrics to new timeline service. Contributed by Junping Du. 2016-07-10 08:45:33 -07:00
Junping Du
d67c9bdb4d YARN-3040. Make putEntities operation be aware of the app's context. Contributed by Zhijie Shen 2016-07-10 08:45:32 -07:00
Junping Du
5e3d9a477b YARN-3034. Implement RM starting its timeline collector. Contributed by Naganarasimha G R 2016-07-10 08:45:32 -07:00
Junping Du
2188a07e5b YARN-3333. Rename TimelineAggregator etc. to TimelineCollector. Contributed by Sangjin Lee 2016-07-10 08:45:31 -07:00
Zhijie Shen
9b56364080 YARN-3039. Implemented the app-level timeline aggregator discovery service. Contributed by Junping Du. 2016-07-10 08:45:31 -07:00
Varun Saxena
5252562edf YARN-5297. Avoid printing a stack trace when recovering an app after the RM restarts. (Junping Du via Varun Saxena). 2016-07-09 00:09:25 +05:30
Junping Du
30ee57ceb1 YARN-4939. The decommissioning Node should keep alive during NM restart. Contributed by sandflee. 2016-07-08 04:14:53 -07:00
Wangda Tan
04f6ebb66a YARN-5294. Pass remote ip address down to YarnAuthorizationProvider. (Jian He via wangda) 2016-07-06 10:36:48 -07:00
Varun Saxena
8e672e3c71 YARN-5286. Add RPC port info in RM web service's response when getting app status. (Jun Gong via Varun Saxena). 2016-07-05 22:56:07 +05:30
Rohith Sharma K S
26b5e6116f YARN-5262. Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat. 2016-06-29 10:08:30 +05:30