big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19,556 Commits 2 Branches 2 Tags 264 MiB
e673dd1d4d
Commit Graph

1522 Commits

Author SHA1 Message Date
Jason Lowe
766544c0b0 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie 2018-04-20 13:22:20 -05:00
Shashikant Banerjee
e4c39f3247
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-19 08:35:38 +09:00
Rohith Sharma K S
241d985467 YARN-8145. yarn rmadmin -getGroups doesn't return updated groups for user. Contributed by Sunil G. 2018-04-18 13:39:41 +05:30
Yufei
d426b980ac YARN-7966. Remove method AllocationConfiguration#getQueueAcl and related unit tests. Contributed by Sen Zhao. 2018-04-17 22:53:32 -07:00
Haibo Chen
bb92bfb4ef YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 11:15:20 -07:00
Haibo Chen
86dfb70875 Revert "YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)" 
This reverts commit 0c14b6f0d8.
 2018-04-17 06:04:38 -07:00
Haibo Chen
0c14b6f0d8 YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 05:59:03 -07:00
Wangda Tan
896b473f1b YARN-8138. Add unit test to validate queue priority preemption works under node partition. (Zian Chen via wangda) 
Change-Id: Ibebfab98a714c12c2dc643b6d7b9754a7f813632
(cherry picked from commit 6ee62e6b1c9b4bc3447ce870446068e626b1a492)
 2018-04-14 11:08:36 -07:00
Eric E Payne
18844599ae YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe 2018-04-12 17:53:57 +00:00
Weiwei Yang
7eb783e263 YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang. 2018-04-11 17:15:25 +08:00
Sunil G
821b0de4c5 YARN-7574. Add support for Node Labels on Auto Created Leaf Queue Template. Contributed by Suma Shivaprasad. 2018-04-09 21:17:22 +05:30
Vrushali C
345e7624d5 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 2018-04-04 15:08:03 -07:00
Konstantinos Karanasos
7853ec8d2f YARN-8013. Support application tags when defining application namespaces for placement constraints. Contributed by Weiwei Yang. 2018-04-04 10:51:58 -07:00
Haibo Chen
93d47a0ed5 YARN-8051. TestRMEmbeddedElector#testCallbackSynchronization is flaky. (Robert Kanter via Haibo Chen) 2018-04-03 07:59:20 -07:00
Wangda Tan
c78cb18c61 YARN-8091. Revisit checkUserAccessToQueue RM REST API. (wangda) 
Change-Id: I5fab3fe229c34e967487b7327c7b3c8ddf7cb795
 2018-04-02 15:22:05 -07:00
Weiwei Yang
1606dad133 YARN-7497. Add file system based scheduler configuration store. Contributed by Jiandan Yang. 2018-03-30 21:41:33 +08:00
Eric Yang
8fb00c3fce YARN-6257. Fix CapacityScheduler REST API JSON output. 
Contributed By Tao Yang
 2018-03-29 17:36:34 -04:00
Sunil G
7a59d60e0c YARN-8085. ResourceProfilesManager should be set in RMActiveServiceContext. Contributed by Tao Yang. 2018-03-29 21:41:16 +05:30
Wangda Tan
47f711eebc YARN-6629. NPE occurred when container allocation proposal is applied but its resource requests are removed before. (Tao Yang via wangda) 
Change-Id: I805880f90b3f6798ec96ed8e8e75755f390a9ad5
 2018-03-28 11:07:45 -07:00
Wangda Tan
a90471b3e6 YARN-8016. Refine PlacementRule interface and add a app-name queue mapping rule as an example. (Zian Chen via wangda) 
Change-Id: I35caf1480e0f76f5f3a53528af09312e39414bbb
 2018-03-23 16:43:40 -07:00
Wangda Tan
a08921ca6c YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (Weiwei Yang via wangda) 
Change-Id: I63b4e4192a95bf7ded98c54e46a2871c72869700
 2018-03-19 11:04:27 -07:00
Sunil G
6bd130ea45 YARN-8028. Support authorizeUserAccessToQueue in RMWebServices. Contributed by Wangda Tan. 2018-03-18 11:00:30 +05:30
Wangda Tan
b167d60763 YARN-7657. Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue. (Suma Shivaprasad via wangda) 
Change-Id: I32d566e8727840e43c0d66e39a77edef017e3a83
 2018-03-13 17:55:51 -07:00
Weiwei Yang
b451889e8e YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Tao Yang. 2018-03-08 18:13:36 +08:00
Arun Suresh
1054b48c27 YARN-7972. Support inter-app placement constraints for allocation tags by application ID. (Weiwei Yang via asuresh) 2018-03-05 11:24:17 -08:00
Arpit Agarwal
2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Yufei Gu
92cbbfe79e YARN-5028. RMStateStore should trim down app state for completed applications. Contributed by Gergo Repas. 2018-02-21 11:42:51 -08:00
Jason Lowe
9497215084 YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:06:28 -06:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Sunil G
aae629913c YARN-7292. Retrospect Resource Profile Behavior for overriding capability. Contributed by Wangda Tan. 2018-02-16 12:37:40 +05:30
Konstantinos Karanasos
0b489e564c YARN-7920. Simplify configuration for PlacementConstraints. Contributed by Wangda Tan. 2018-02-15 14:23:38 -08:00
Eric Payne
bddfe42e2c Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit c5e6e3de1c.
 2018-02-14 14:29:20 -06:00
Wangda Tan
042ef2fa7b YARN-7789. Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator. (Zian Chen via wangda) 
Change-Id: I1f6a451fe16758def6f47c046a9b8a67ad7255af
 2018-02-14 23:11:10 +08:00
Eric Payne
c5e6e3de1c YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-13 10:11:02 -06:00
Wangda Tan
d02e42cee4 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
 2018-02-12 10:29:37 +08:00
Yufei Gu
1bc03ddf97 YARN-7655. Avoid AM preemption caused by RRs for specific nodes or racks. Contributed by Steven Rand. 2018-02-08 12:32:43 -08:00
Robert Kanter
814d701d46 YARN-7841. Cleanup AllocationFileLoaderService's reloadAllocations method (snemeth via rkanter) 2018-02-06 14:36:49 -08:00
Weiwei Yang
bff858e910 YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 2018-02-06 19:33:04 +08:00
Konstantinos Karanasos
50723889cc YARN-7778. Merging of placement constraints defined at different levels. Contributed by Weiwei Yang. 2018-02-02 14:46:20 -08:00
Arun Suresh
6e5ba9366f YARN-7839. Modify PlacementAlgorithm to Check node capacity before placing request on node. (Panagiotis Garefalakis via asuresh) 2018-02-02 10:28:22 -08:00
Arun Suresh
d481344783 YARN-7822. Constraint satisfaction checker support for composite OR and AND constraints. (Weiwei Yang via asuresh) 2018-01-31 01:51:08 -08:00
Arun Suresh
add993e26a YARN-6597. Add RMContainer recovery test to verify tag population in the AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f8c5f5b237 YARN-7784. Fix Cluster metrics when placement processor is enabled. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a4c539fcdb YARN-7783. Add validation step to ensure constraints are not violated due to order in which a request is processed. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos
adbe87abf8 YARN-7788. Factor out management of temp tags from AllocationTagsManager. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Arun Suresh
8bf7c44436 YARN-7763. Allow Constraints specified in the SchedulingRequest to override application level constraints. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
28fe7f3318 YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
bdba01f73b YARN-7682. Expose canSatisfyConstraints utility function to validate a placement against a constraint. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
1efb2b6f25 YARN-6596. Introduce Placement Constraint Manager module. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
37f1a7b64f YARN-7653. Node group support for AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
88d8d3f40b YARN-7670. Modifications to the ResourceScheduler API to support SchedulingRequests. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
801c0988b5 YARN-7522. Introduce AllocationTagsManager to associate allocation tags to nodes. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
e9c72d04be YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 2018-01-29 20:44:38 +05:30
Wangda Tan
75618c0dec YARN-7760. [UI2] Clicking 'Master Node' or link next to 'AM Node Web UI' under application's appAttempt page goes to OLD RM UI. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ia7e2b955d3551ca43c6852fe253fd01983b56565
 2018-01-26 15:44:55 +08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Sunil G
82cc6f6968 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 2018-01-25 14:32:54 +05:30
Yufei Gu
22ee6f77e5 YARN-7755. Clean up deprecation messages for allocation increments in FS config. Contributed by Wilfred Spiegelenburg. 2018-01-22 10:30:01 -08:00
Miklos Szegedi
bc93ac229e YARN-7139. FairScheduler: finished applications are always restored to default queue. Contributed by Wilfred Spiegelenburg. 2018-01-18 17:43:47 -08:00
Sunil G
06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Yufei Gu
370f1c6283 YARN-6486. FairScheduler: Deprecate continuous scheduling. (Contributed by Wilfred Spiegelenburg) 2018-01-16 14:46:55 -08:00
Rohith Sharma K S
d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Sunil G
afd8caba27 YARN-7722. Rename variables in MockNM, MockRM for better clarity. Contributed by Lovekesh bansal 2018-01-10 20:51:34 +05:30
Jason Lowe
90c86d6e09 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 2018-01-09 10:51:15 -06:00
Jason Lowe
ebff4decfe YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 2018-01-09 09:52:59 -06:00
Ray Chiang
59ab5da0a0 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2018-01-08 16:19:02 -08:00
Wangda Tan
a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter
2aa4f0a559 YARN-7645. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers is flakey with FairScheduler (rkanter) 2018-01-05 13:55:09 -08:00
Robert Kanter
f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Sunil G
0c75d0634b YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:42:17 +05:30
Robert Kanter
7a55044803 YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-03 15:31:50 -08:00
Robert Kanter
382215c72b YARN-7577. Unit Fail: TestAMRestart#testPreemptedAMRestartOnRMRestart (miklos.szegedi@cloudera.com via rkanter) 2017-12-20 13:39:00 -08:00
Eric Yang
94a2ac6b71 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 18:42:27 -05:00
Sunil G
890d3d0645 YARN-7638. Unit tests related to preemption for auto created leaf queues feature.Contributed by Suma Shivaprasad. 2017-12-15 13:00:57 +05:30
Sunil G
cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Sunil G
8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G
5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G
312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Wangda Tan
a8316df8c0 YARN-7520. Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues. (Suma Shivaprasad via wangda) 
Change-Id: I482f086945bd448d512cb5b3879d7371e37ee134
 2017-12-08 15:11:28 -08:00
Wangda Tan
b38643c9a8 YARN-7473. Implement Framework and policy for capacity management of auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: Icca7805fe12f6f7fb335effff4b121b6f7f6337b
 2017-12-08 15:10:16 -08:00
Wangda Tan
74665e3a7d YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
 2017-12-08 15:07:56 -08:00
Sunil G
4db4a4a165 YARN-7575. NPE in scheduler UI when max-capacity is not configured. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G
daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan
1012b901c8 YARN-7544. Use queue-path.capacity/maximum-capacity to specify absolute min/max resources. (Sunil G via wangda) 
Change-Id: I685341be213eee500f51e02f01c91def89391c17
 2017-12-07 18:56:54 -08:00
Wangda Tan
b7b8cd5324 YARN-7538. Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring. (Sunil G via wangda) 
Change-Id: Ic9bd7e599c56970fe01cb0e1bba6df7d1f77eb29
 2017-12-07 18:56:54 -08:00
Wangda Tan
7462c38277 YARN-7483. CapacityScheduler test cases cleanup post YARN-5881. (Sunil G via wangda) 
Change-Id: I9741a6baf5cb7352d05636efb6c0b24790e7589a
 2017-12-07 18:56:54 -08:00
Wangda Tan
034b312d9f YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
 2017-12-07 18:56:54 -08:00
Wangda Tan
aa3f62740f YARN-7332. Compute effectiveCapacity per each resource vector. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda) 
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
 2017-12-07 18:56:54 -08:00
Sunil G
a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G
f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Robert Kanter
d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00