big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19,576 Commits 2 Branches 2 Tags 264 MiB
d70d845705
Commit Graph

4677 Commits

Author SHA1 Message Date
Sunil G
64344c345d YARN-7802. [UI2] Application regex search did not work properly with app name. Contributed by Sreenath Somarajapuram. 2018-01-31 17:40:36 +05:30
Arun Suresh
8df7666fe1 YARN-7780. Documentation for Placement Constraints. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
add993e26a YARN-6597. Add RMContainer recovery test to verify tag population in the AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f8c5f5b237 YARN-7784. Fix Cluster metrics when placement processor is enabled. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
644afe5fd8 YARN-7807. Assume intra-app anti-affinity as default for scheduling request inside AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a4c539fcdb YARN-7783. Add validation step to ensure constraints are not violated due to order in which a request is processed. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos
adbe87abf8 YARN-7788. Factor out management of temp tags from AllocationTagsManager. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Arun Suresh
8bf7c44436 YARN-7763. Allow Constraints specified in the SchedulingRequest to override application level constraints. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
28fe7f3318 YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) 2018-01-31 01:30:17 -08:00
Wangda Tan
e60f51299d YARN-7745. Allow DistributedShell to take a placement specification for containers it wants to launch. (Arun Suresh via wangda) 
Change-Id: Ided146d662e944a8a4692e5d6885f23fd9bbcad5
 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
8779a35742 YARN-7709. Remove SELF from TargetExpression type. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Wangda Tan
29d9e4d581 YARN-6619. AMRMClient Changes to use the PlacementConstraint and SchcedulingRequest objects. (Arun Suresh via wangda) 2018-01-31 01:30:17 -08:00
Arun Suresh
a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
4eda58c136 YARN-7681. Double-check placement constraints in scheduling phase before actual allocation is made. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
bdba01f73b YARN-7682. Expose canSatisfyConstraints utility function to validate a placement against a constraint. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
1efb2b6f25 YARN-6596. Introduce Placement Constraint Manager module. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
37f1a7b64f YARN-7653. Node group support for AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
88d8d3f40b YARN-7670. Modifications to the ResourceScheduler API to support SchedulingRequests. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
801c0988b5 YARN-7522. Introduce AllocationTagsManager to associate allocation tags to nodes. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
69de9a1ba9 YARN-7448. [API] Add SchedulingRequest to the AllocateRequest. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos
db928556c8 YARN-6595. [API] Add Placement Constraints at the application level. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Wangda Tan
b57e8bc300 YARN-6594. [API] Introduce SchedulingRequest object. (Konstantinos Karanasos via wangda) 2018-01-31 01:30:17 -08:00
Wangda Tan
33a796d9b7 YARN-6593. [API] Introduce Placement Constraint object. (Konstantinos Karanasos via wangda) 
Change-Id: Id00edb7185fdf01cce6e40f920cac3585f8cbe9c
 2018-01-31 01:30:17 -08:00
Eric Yang
f9dd5b61f4 YARN-7811. Fixed a bug in user defined docker network settings. (Contributed by Billie Rinaldi) 2018-01-30 12:42:38 -05:00
Jason Lowe
901d15a30b Revert "YARN-2185. Use pipes when localizing archives. Contributed by Miklos Szegedi" 
This reverts commit 1b0f265db1.
 2018-01-30 08:34:39 -06:00
Sunil G
6463e10c72 YARN-7723. Avoid using docker volume --format option to run against to older docker releases. Contributed by Wangda Tan 2018-01-30 15:58:11 +05:30
Wangda Tan
f666e7c43d YARN-5148. [UI2] Add page to new YARN UI to view server side configurations/logs/JVM-metrics. (Kai Sasaki/Sunil G via wangda) 
Change-Id: I5de88ce9850c0bc337dcb2c7d25ee9ad52016925
 2018-01-30 15:45:06 +08:00
Sunil G
e9c72d04be YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 2018-01-29 20:44:38 +05:30
Jinjiang Ling
3400d0c535
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-30 00:00:57 +09:00
Jian He
443523f9c0 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:18:51 -08:00
Haibo Chen
649ef7ac33 YARN-7064. Use cgroup to get container resource utilization. (Miklos Szegedi via Haibo Chen) 2018-01-26 16:27:31 -08:00
Jason Lowe
1b0f265db1 YARN-2185. Use pipes when localizing archives. Contributed by Miklos Szegedi 2018-01-26 13:25:20 -06:00
Billie Rinaldi
f2fa736f0a YARN-7797. Docker host network can not obtain IP address for RegistryDNS. Contributed by Eric Yang 2018-01-26 09:32:23 -08:00
Wangda Tan
8b5b045bd2 YARN-7742. [UI2] Duplicated containers are rendered per attempt. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ib2bb3c64d7f09a7c3488f498d0079bcd2e6ac023
 2018-01-26 15:46:06 +08:00
Wangda Tan
75618c0dec YARN-7760. [UI2] Clicking 'Master Node' or link next to 'AM Node Web UI' under application's appAttempt page goes to OLD RM UI. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ia7e2b955d3551ca43c6852fe253fd01983b56565
 2018-01-26 15:44:55 +08:00
Wangda Tan
e0cfb0a31a YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
 2018-01-26 15:43:27 +08:00
Miklos Szegedi
2e5865606b YARN-7814. Remove automatic mounting of the cgroups root directory into Docker containers. Contributed by Shane Kumpf. 2018-01-25 17:37:23 -08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Sunil G
82cc6f6968 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 2018-01-25 14:32:54 +05:30
Eric Yang
59828be197 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)
 2018-01-24 21:24:19 -05:00
Wangda Tan
051ad9d63f YARN-7628. [Documentation] Documenting the ability to disable elasticity at Capacity Scheduler's sub queues. (Zian Chen via wangda) 
Change-Id: I5c37df1c5bee9e3a78379775ef307ef94506ebc4
 2018-01-25 07:24:57 +08:00
Billie Rinaldi
0c559b2782 YARN-7777. Fix user name format in YARN Registry DNS name. Contributed by Jian He 2018-01-24 13:50:47 -08:00
Zhe Zhang
fa8cf4d1b4 YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 2018-01-24 12:54:54 -08:00
Jian He
55c32776b1 YARN-7801. AmFilterInitializer should addFilter after fill all parameters. Contributed by Wangda Tan 2018-01-24 11:17:12 -08:00
Sunil G
97607617ce YARN-7806. Distributed Shell should use timeline async api's. Contributed by Rohith Sharma K S 2018-01-24 16:56:13 +05:30
Rohith Sharma K S
2014863470 YARN-7749. [UI2] GPU information tab in left hand side disappears when we click other tabs below. Contributed by Sunil G and Vasudevan Skm. 2018-01-24 15:56:59 +05:30
Rohith Sharma K S
f3e33aeb80 Revert "YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma" 
This reverts commit ec8f47e7fa.
 2018-01-24 15:04:15 +05:30
Miklos Szegedi
e7642a3e6f YARN-7796. Container-executor fails with segfault on certain OS configurations. Contributed by Gergo Repas. 2018-01-23 21:07:05 -08:00
Billie Rinaldi
e307edcb47 YARN-7540 and YARN-7605. Convert yarn app cli to call yarn api services and implement doAs for Api Service REST API. Contributed by Eric Yang and Jian He 2018-01-23 17:54:39 -08:00
Jian He
39b999aba2 YARN-7782. Enable user re-mapping for Docker containers in yarn-default.xml. Contributed by Eric Yang 2018-01-23 14:03:53 -08:00
Jian He
a72cdcc47a YARN-7766. Introduce a new config property for YARN Service dependency tarball location. Contributed by Gour Saha 2018-01-23 10:53:27 -08:00
Eric Yang
97fe3cc187 YARN-7729. Add support for setting Docker PID namespace mode. (Contributed by Billie Rinaldi) 2018-01-22 16:33:38 -05:00
Yufei Gu
22ee6f77e5 YARN-7755. Clean up deprecation messages for allocation increments in FS config. Contributed by Wilfred Spiegelenburg. 2018-01-22 10:30:01 -08:00
Vrushali C
ec8f47e7fa YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma 2018-01-19 15:34:40 -08:00
Haibo Chen
4aca4ff759 YARN-5094. some YARN container events have timestamp of -1. 2018-01-19 10:27:46 -08:00
Rohith Sharma K S
c5bbd6418e YARN-7753. [UI2] Application logs has to be pulled from ATS 1.5 instead of ATS2. Contributed by Sunil G. 2018-01-19 15:48:16 +05:30
Miklos Szegedi
bc93ac229e YARN-7139. FairScheduler: finished applications are always restored to default queue. Contributed by Wilfred Spiegelenburg. 2018-01-18 17:43:47 -08:00
Billie Rinaldi
37f4696a9c YARN-7740. Fix logging for destroy yarn service cli when app does not exist and some minor bugs. Contributed by Jian He 2018-01-18 12:11:19 -08:00
Sunil G
06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Sunil G
8e5472b1e6 YARN-7750. [UI2] Render time related fields in all pages to the browser timezone. Contributed by Vasudevan Skm. 2018-01-18 18:54:41 +05:30
Miklos Szegedi
a68e445dc6 YARN-7717. Add configuration consistency for module.enabled and docker.privileged-containers.enabled. Contributed by Eric Badger. 2018-01-17 14:11:14 -08:00
Miklos Szegedi
41049ba5d1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-16 15:40:43 -08:00
Yufei Gu
370f1c6283 YARN-6486. FairScheduler: Deprecate continuous scheduling. (Contributed by Wilfred Spiegelenburg) 2018-01-16 14:46:55 -08:00
Robert Kanter
5ac109909a YARN-7479. TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk (ajisakaa via rkanter) 2018-01-16 10:16:42 -08:00
Rohith Sharma K S
d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Miklos Szegedi
2dcfc1876e YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 2018-01-12 17:00:01 -08:00
Wangda Tan
edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Billie Rinaldi
53f2768926 YARN-7724. yarn application status should support application name. Contributed by Jian He 2018-01-12 15:33:46 -08:00
Jian He
c05b84703b YARN-7671. Improve Diagonstic message for stop yarn native service. Contributed by Chandni Singh 2018-01-12 14:04:15 -08:00
Eric Yang
3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Billie Rinaldi
4fb1f45f21 YARN-7731. RegistryDNS should handle upstream DNS returning CNAME. Contributed by Eric Yang 2018-01-12 09:21:17 -08:00
Sunil G
128d773a23 YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 2018-01-12 15:41:19 +05:30
Miklos Szegedi
bc285da107 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 2018-01-11 09:40:09 -08:00
Akira Ajisaka
fbbbf59c82
YARN-7735. Fix typo in YARN documentation. Contributed by Takanobu Asanuma. 2018-01-11 15:59:27 +09:00
Sunil G
afd8caba27 YARN-7722. Rename variables in MockNM, MockRM for better clarity. Contributed by Lovekesh bansal 2018-01-10 20:51:34 +05:30
Miklos Szegedi
55066cc53d YARN-7689. TestRMContainerAllocator fails after YARN-6124. Contributed by Wilfred Spiegelenburg. 2018-01-09 16:59:39 -08:00
Haibo Chen
8ee7080e5d YARN-7665. Allow FS scheduler state dump to be turned on/off separately from FS debug log. (Wilfred Spiegelenburg via Haibo Chen) 2018-01-09 16:45:35 -08:00
Haibo Chen
f725b9e267 YARN-7716. metricsTimeStart and metricsTimeEnd should be all lower case in the doc. 2018-01-09 08:56:59 -08:00
Jason Lowe
90c86d6e09 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 2018-01-09 10:51:15 -06:00
Jason Lowe
ebff4decfe YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 2018-01-09 09:52:59 -06:00
bibinchundatt
b26e30ab16 YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 2018-01-09 17:57:01 +05:30
Sunil G
783a01eb4a YARN-7718. DistributedShell failed to specify resource other than memory/vcores from container_resources. Contributed by Wangda Tan. 2018-01-09 16:39:33 +05:30
Ray Chiang
59ab5da0a0 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2018-01-08 16:19:02 -08:00
Billie Rinaldi
dc54747d70 YARN-7704. Document improvement for registry dns. Contributed by Jian He 2018-01-08 13:20:17 -08:00
Sunil G
01f3f2167e YARN-7242. Support to specify values of different resource types in DistributedShell for easier testing. Contributed by Gergely Novák. 2018-01-08 11:59:06 +05:30
Rohith Sharma K S
c2d6fa3656 YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 2018-01-08 10:39:31 +05:30
Jian He
836e3c45e8 Revert "YARN-7540. Route YARN service CLI function through YARN Service API. (Contributed by Eric Yang)" 
This reverts commit 438c1d333e.
 2018-01-05 18:25:00 -08:00
Wangda Tan
a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter
2aa4f0a559 YARN-7645. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers is flakey with FairScheduler (rkanter) 2018-01-05 13:55:09 -08:00
Robert Kanter
f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Sunil G
0c75d0634b YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:42:17 +05:30
Jason Lowe
d795661868 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:15:52 -06:00
Robert Kanter
7a55044803 YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-03 15:31:50 -08:00
Arpit Agarwal
4379113bda HADOOP-15093. Deprecation of yarn.resourcemanager.zk-address is undocumented. Contributed by Ajay Kumar. 2018-01-03 14:32:47 -08:00
Haibo Chen
2f6c038be6 YARN-7602. NM should reference the singleton JvmMetrics instance. 2018-01-03 09:41:26 -08:00
Rohith Sharma K S
c9bf813c9a YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 2018-01-03 18:20:04 +05:30
Arun Suresh
c0c7cce81d YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 2018-01-02 22:03:00 -08:00
Miklos Szegedi
7f515f57ed YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 2018-01-02 18:03:04 -08:00
Miklos Szegedi
80440231d4 YARN-6894. RM Apps API returns only active apps when query parameter queue used. Contributed by Gergely Novák. 2018-01-02 17:59:10 -08:00
Miklos Szegedi
626b5103d4 YARN-7688. Miscellaneous Improvements To ProcfsBasedProcessTree. Contributed by BELUGA BEHR. 2018-01-02 17:39:50 -08:00
Miklos Szegedi
33ae2a4ae1 YARN-7687. ContainerLogAppender Improvements. Contributed by BELUGA BEHR. 2018-01-02 16:36:51 -08:00
Wangda Tan
7467e8fe5a YARN-7555. Support multiple resource types in YARN native services. (wangda) 
Change-Id: I330e6ee17a73962dcaadd766a3e72d2888681731
 2017-12-29 15:34:08 -08:00
Miklos Szegedi
b82049b4f0 YARN-7580. ContainersMonitorImpl logged message lacks detail when exceeding memory limits. Contributed by Wilfred Spiegelenburg. 2017-12-29 12:49:37 -08:00
Arun Suresh
a55884c68e YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 2017-12-28 22:20:42 -08:00
Robert Kanter
382215c72b YARN-7577. Unit Fail: TestAMRestart#testPreemptedAMRestartOnRMRestart (miklos.szegedi@cloudera.com via rkanter) 2017-12-20 13:39:00 -08:00
Rohith Sharma K S
13ad7479b0 YARN-7674. Update Timeline Reader web app address in UI2. Contributed by Sunil G. 2017-12-20 22:28:28 +05:30
Sunil G
d62932c3b2 YARN-7032. [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Contributed by Rohith Sharma K S. 2017-12-20 11:31:15 +05:30
Eric Yang
41b581012a YARN-7616. Map YARN application status to Service Status more accurately. (Contributed by Gour Saha) 2017-12-19 19:14:45 -05:00
Eric Yang
94a2ac6b71 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 18:42:27 -05:00
Eric Yang
989c75109a YARN-7543. Add check for max cpu limit and missing file for YARN service. 
(Contributed by Jian He)
 2017-12-19 16:45:04 -05:00
Varun Saxena
c0aeb666a4 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 2017-12-19 22:29:24 +05:30
Sunil G
fe5b057c81 YARN-7620. Allow node partition filters on Queues page of new YARN UI. Contributed by Vasudevan Skm. 2017-12-19 20:27:25 +05:30
Junping Du
a7f8caf58e Add 2.8.3 release jdiff files. 
(cherry picked from commit c89f99aade575ab1f6a9836df719cda272293d90)
 2017-12-18 22:39:01 -08:00
Jason Lowe
811fabdebe YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 2017-12-18 15:20:06 -06:00
Akira Ajisaka
001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Weiwei Yang
9289641020 YARN-7617. Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are started. Contributed by Weiwei Yang. 2017-12-18 10:07:16 +08:00
Wangda Tan
44825f0960 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
 2017-12-15 15:40:56 -08:00
Wangda Tan
631b5c2db7 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
 2017-12-15 15:38:36 -08:00
Sunil G
6681dd1007 YARN-7633. Documentation for auto queue creation feature and related configurations. Contributed by Suma Shivaprasad. 2017-12-15 23:07:15 +05:30
Rohith Sharma K S
09d996fdd4 YARN-7190. Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath. Contributed by Varun Saxena. 2017-12-15 21:50:28 +05:30
Sunil G
890d3d0645 YARN-7638. Unit tests related to preemption for auto created leaf queues feature.Contributed by Suma Shivaprasad. 2017-12-15 13:00:57 +05:30
Weiwei Yang
89b6c482c1 YARN-7642. Add test case to verify context update after container promotion or demotion with or without auto update. Contributed by Weiwei Yang. 2017-12-15 11:58:52 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
95d4ec7fc0 YARN-7565. Addendum to fix an incompatible change. Contributed by Wangda Tan. 2017-12-14 17:10:37 -08:00
Subru Krishnan
17ba74be29 YARN-7630. Fix AMRMToken rollover handling in AMRMProxy. Contributed by Botong Huang. 2017-12-14 14:03:55 -08:00
Sunil G
2564b4d07f YARN-7119. Support multiple resource types in rmadmin updateNodeResource command. Contributed by Manikandan R. 2017-12-14 16:11:57 +05:30
Eric Yang
438c1d333e YARN-7540. Route YARN service CLI function through YARN Service API. (Contributed by Eric Yang) 2017-12-13 19:29:04 -05:00
Chen Liang
46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Sunil G
cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Sunil G
10fc8d2a7d YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 2017-12-13 22:00:07 +05:30
Sunil G
880cd754a5 YARN-7536. em-table improvement for better filtering in new YARN UI. Contributed by Vasudevan Skm. This closes #313. 2017-12-13 16:46:12 +05:30
Sunil G
fa4ecd8b98 YARN-7641. Allow searchable filter for Application page log viewer in new YARN UI. Contributed by Vasudevan Skm. This closes #312. 2017-12-13 15:51:49 +05:30
Weiwei Yang
7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe
2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jian He
3ebe6a7819 YARN-7565. Yarn service pre-maturely releases the container after AM restart. Contributed by Chandni Singh 2017-12-12 13:35:56 -08:00
Jason Lowe
06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Sunil G
8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G
5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G
312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Weiwei Yang
a2edc4cbf5 YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:31:46 +08:00
Subru Krishnan
670e8d4ec7 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 2017-12-08 15:39:18 -08:00
Wangda Tan
04b84da245 YARN-7443. Add native FPGA module support to do isolation with cgroups. (Zhankun Tang via wangda) 
Change-Id: Ic4b7f9f3e032986b8f955139c9fe4d3a6c818a53
 2017-12-08 15:18:22 -08:00
Wangda Tan
adca1a72e4 YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I46689e530550ee0a6ac7a29786aab2cc1bdf314f
 2017-12-08 15:17:02 -08:00
Wangda Tan
a8316df8c0 YARN-7520. Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues. (Suma Shivaprasad via wangda) 
Change-Id: I482f086945bd448d512cb5b3879d7371e37ee134
 2017-12-08 15:11:28 -08:00
Wangda Tan
f548bfffbd YARN-7420. YARN UI changes to depict auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: I8039d3772a191ddede132cd1f8b08a8ca2e275b7
 2017-12-08 15:10:47 -08:00
Wangda Tan
b38643c9a8 YARN-7473. Implement Framework and policy for capacity management of auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: Icca7805fe12f6f7fb335effff4b121b6f7f6337b
 2017-12-08 15:10:16 -08:00
Wangda Tan
74665e3a7d YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
 2017-12-08 15:07:56 -08:00
Sunil G
532bbf4602 YARN-7533. Documentation for absolute resource support in Capacity Scheduler. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G
4db4a4a165 YARN-7575. NPE in scheduler UI when max-capacity is not configured. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G
daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan
1012b901c8 YARN-7544. Use queue-path.capacity/maximum-capacity to specify absolute min/max resources. (Sunil G via wangda) 
Change-Id: I685341be213eee500f51e02f01c91def89391c17
 2017-12-07 18:56:54 -08:00
Wangda Tan
b7b8cd5324 YARN-7538. Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring. (Sunil G via wangda) 
Change-Id: Ic9bd7e599c56970fe01cb0e1bba6df7d1f77eb29
 2017-12-07 18:56:54 -08:00
Wangda Tan
7462c38277 YARN-7483. CapacityScheduler test cases cleanup post YARN-5881. (Sunil G via wangda) 
Change-Id: I9741a6baf5cb7352d05636efb6c0b24790e7589a
 2017-12-07 18:56:54 -08:00
Rohith Sharma K S
e65ca92fb6 YARN-7482. Max applications calculation per queue has to be retrospected with absolute resource support. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan
034b312d9f YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
 2017-12-07 18:56:54 -08:00
Wangda Tan
aa3f62740f YARN-7332. Compute effectiveCapacity per each resource vector. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda) 
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
 2017-12-07 18:56:54 -08:00
Weiwei Yang
e411dd6666 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:29:40 +08:00
Weiwei Yang
40b0045ebe YARN-7610. Extend Distributed Shell to support launching job with opportunistic containers. Contributed by Weiwei Yang. 2017-12-06 17:52:41 +08:00
Weiwei Yang
05c347fe51 YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 2017-12-06 12:21:52 +08:00
Wangda Tan
0311cf0535 YARN-7381. Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml. (Xuan Gong via wangda) 
Change-Id: I1ed58dafad5cc276eea5c0b0813cf04f57d73a87
 2017-12-05 13:09:49 -08:00
Sunil G
a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G
99ccca341f YARN-7092. Render application specific log under application tab in new YARN UI. Contributed by Akhil PB. 2017-12-05 19:41:07 +05:30
Sunil G
f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Arpit Agarwal
e00c7f78c1 HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts. 2017-12-04 21:02:04 -08:00
Robert Kanter
d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Eric Yang
d30d57828f YARN-6669. Implemented Kerberos security for YARN service framework. (Contributed by Jian He) 2017-12-04 15:11:00 -05:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Sunil G
81f6e46b2f YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 2017-12-04 11:27:23 +05:30
Sunil G
30f2646b15 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2017-12-04 10:45:07 +05:30
Jason Lowe
60f95fb719 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 2017-12-01 15:47:01 -06:00
Jason Lowe
25df505421 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 2017-12-01 15:37:36 -06:00
Robert Kanter
c83fe44917 YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 2017-12-01 12:18:13 -08:00
Wangda Tan
7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Sunil G
556aea3f36 YARN-7487. Ensure volume to include GPU base libraries after created by plugin. Contributed by Wangda Tan. 2017-12-01 13:36:28 +05:30
Sunil G
4653aa3eb3 YARN-7546. Layout changes in Queue UI to show queue details on right pane. Contributed by Vasudevan Skm. 2017-12-01 13:26:01 +05:30
Wangda Tan
a63d19d365 YARN-6124. Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues. (Zian Chen via wangda) 
Change-Id: Id93656f3af7dcd78cafa94e33663c78d410d43c2
 2017-11-30 15:57:22 -08:00
Junping Du
a409425986 YARN-7558. yarn logs command fails to get logs for running containers if UI authentication is enabled. Contributed by Xuan Gong. 2017-11-30 13:47:47 -08:00
Jason Lowe
5cfaee2e6d YARN-7495. Improve robustness of the AggregatedLogDeletionService. Contributed by Jonathan Eagles 2017-11-30 12:39:18 -06:00
Wangda Tan
c9a54aab6b YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2017-11-29 17:46:16 -08:00
Wei Yan
333ef303ff YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat. 2017-11-29 14:38:07 -08:00
Daniel Templeton
8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Jason Lowe
a2c7a73e33 YARN-6647. RM can crash during transitionToStandby due to InterruptedException. Contributed by Bibin A Chundatt 2017-11-28 11:15:44 -06:00
Sunil G
6b76695f88 YARN-7480. Render tooltips on columns where text is clipped in new YARN UI. Contributed by Vasudevan Skm. This closes #293 2017-11-28 22:41:52 +05:30
Sunil G
641ba5c7a1 YARN-7499. Layout changes to Application details page in new YARN UI. Contributed by Vasudevan Skm. 2017-11-28 18:37:11 +05:30
Yufei Gu
d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Jian He
fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Yufei Gu
2bde3aedf1 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 2017-11-24 23:32:46 -08:00
Wangda Tan
834e91ee91 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
 2017-11-23 19:59:03 -08:00
Arun Suresh
b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Sunil G
aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Yufei Gu
4cc9479dae YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 2017-11-22 14:18:36 -08:00
Eric Yang
d42a336cfa YARN-5534. Allow user provided Docker volume mount list. (Contributed by Shane Kumpf) 2017-11-22 13:05:34 -05:00
Steve Loughran
de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
yufei
03c311eae3 YARN-7513. Remove the scheduler lock in FSAppAttempt.getWeight() (Contributed by Wilfred Spiegelenburg) 2017-11-21 10:33:34 -08:00
Haibo Chen
67bbbe1c0c YARN-7531. ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType(). 2017-11-21 09:09:16 -08:00
Wangda Tan
0d781dd03b YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
 2017-11-20 11:48:15 -08:00
Billie Rinaldi
6f9d7a146d YARN-7529. TestYarnNativeServices#testRecoverComponentsAfterRMRestart() fails intermittently. Contributed by Chandni Singh 2017-11-20 07:37:04 -08:00
bibinchundatt
b5b81a4f08 YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 2017-11-18 19:25:29 +05:30
bibinchundatt
96e6a99360 YARN-7525. Incorrect query parameters in cluster nodes REST API document. Contributed by Tao Yang. 2017-11-18 19:02:11 +05:30
Subru Krishnan
d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Eric Yang
0940e4f692 YARN-7218. Decouple YARN Services REST API namespace from RM. (Contributed by Eric Yang) 2017-11-17 12:28:12 -05:00
Varun Vasudev
5f0b238a11 YARN-7430. Enable user re-mapping for Docker containers by default. Contributed by Eric Yang. 2017-11-17 12:04:47 +05:30
Wangda Tan
0987a7b8cb YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue mapping. (Suma Shivaprasad via wangda) 
Change-Id: Ia1704bb8cb5070e5b180b5a85787d7b9ca57ebc6
 2017-11-16 11:25:52 -08:00
Jian He
6bf2c30192 YARN-7503. Configurable heap size / JVM opts in service AM. Contributed by Jonathan Hung 2017-11-16 10:53:55 -08:00
Haibo Chen
28d0fcbef4 YARN-7390. All reservation related test cases failed when TestYarnClient runs against Fair Scheduler. (Yufei Gu via Haibo Chen) 2017-11-16 10:48:24 -08:00
Sunil G
61ace174cd YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 2017-11-16 22:34:23 +05:30
Billie Rinaldi
f4d5d20286 YARN-7486. Race condition in service AM that can cause NPE. Contributed by Jian He 2017-11-16 07:58:06 -08:00
Sunil G
09a1342608 YARN-7492. Set up SASS for new YARN UI styling. Contributed by Vasudevan Skm. 2017-11-16 13:44:21 +05:30
Jason Lowe
fac72eef23 YARN-7361. Improve the docker container runtime documentation. Contributed by Shane Kumpf 2017-11-15 12:32:02 -06:00
Daniel Templeton
b246c54749 YARN-7414. FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() 
(Contributed by Soumabrata Chakraborty via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Daniel Templeton
e094eb74b9 YARN-6953. Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() 
(Contributed by Manikandan R via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Sunil G
2f379d412b YARN-7464. Introduce filters in Nodes page of new YARN UI. Contributed by Vasudevan Skm. 2017-11-15 14:02:53 +05:30
Sunil G
c4c57b80e1 YARN-7462. Render outstanding resource requests on application page of new YARN UI. Contributed by Vasudevan Skm. 2017-11-15 12:01:29 +05:30
Jian He
4f40cd314a YARN-7488. Make ServiceClient.getAppId method public to return ApplicationId for a service name. Contributed by Gour Saha 2017-11-13 18:57:56 -08:00
Jian He
5323b0048b YARN-7466. ResourceRequest has a different default for allocationRequestId than Container. Contributed by Chandni Singh 2017-11-13 15:37:39 -08:00
Junping Du
e14f03dfbf YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 15:27:37 -08:00
Daniel Templeton
040a38dc49 YARN-7369. Improve the resource types docs 2017-11-13 11:05:07 -08:00
Daniel Templeton
2e512f016e YARN-7442. [YARN-7069] Limit format of resource type name (Contributed by Wangda Tan via Daniel Templeton) 2017-11-13 11:03:30 -08:00
Eric Yang
fa4b5c669c YARN-7447. Fixed bug in create YARN services via RM. (Contributed by Billie Rinaldi) 2017-11-13 13:59:58 -05:00
Sunil G
fb62bd625f YARN-7445. Render Applications and Services page with filters in new YARN UI. Contributed by Vasudevan Skm. 2017-11-13 19:41:49 +05:30
Subru Krishnan
3e26077848 YARN-7475. Fix Container log link in new YARN UI. (Sunil G via Subru). 
(cherry picked from commit 3c5b46c2edd69bb238d635ae61ff91656dec23df)
 2017-11-12 09:53:39 -08:00
Sunil G
ff9f7fcf7f YARN-7452. Decommissioning node default value to be zero in new YARN UI. Contributed by Vasudevan Skm. 2017-11-11 16:52:31 +05:30
bibinchundatt
2c2b7a3672 YARN-7406. Moving logging APIs over to slf4j in hadoop-yarn-api. Contributed by Yeliang Cang. 2017-11-11 10:36:27 +05:30
Wangda Tan
dd07038ffa YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-09 14:51:15 -08:00
Sean Mackrory
1883a00249 YARN-7465. start-yarn.sh fails to start ResourceManagers unless running as root. 2017-11-09 14:47:43 -07:00
Konstantinos Karanasos
ac4d2b1081 YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 2017-11-09 13:01:24 -08:00
Robert Kanter
a2c150a736 YARN-7386. Duplicate Strings in various places in Yarn memory (misha@cloudera.com via rkanter) 2017-11-09 12:12:52 -08:00
Daniel Templeton
462f6c490e YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
 2017-11-09 11:58:49 -08:00
Haibo Chen
a1382a18df YARN-7388. TestAMRestart should be scheduler agnostic. 2017-11-09 10:49:50 -08:00
bibinchundatt
0a72c2f56c YARN-7454. RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup. Contributed by Jason Lowe. 2017-11-09 21:01:19 +05:30
Daniel Templeton
49b4c0b334 YARN-7458. TestContainerManagerSecurity is still flakey 
(Contributed by Robert Kanter via Daniel Templeton)

Change-Id: Ibb1975ad086c3a33f8af0b4f8b9a13c3cdca3f7d
 2017-11-08 17:31:14 -08:00
Daniel Templeton
0de10680b7 YARN-7166. Container REST endpoints should report resource types 
Change-Id: If9c2fe58d4cf758bb6b6cf363dc01f35f8720987
 2017-11-08 16:43:49 -08:00
Arun Suresh
cb35a59589 YARN-7343. Add a junit test for ContainerScheduler recovery. (Sampada Dehankar via asuresh) 2017-11-08 08:14:02 -08:00
Arun Suresh
a9c70b0e84 YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 2017-11-08 08:00:53 -08:00
Daniel Templeton
8db9d61ac2 YARN-7401. Reduce lock contention in ClusterNodeTracker#getClusterCapacity() 2017-11-07 14:53:48 -08:00
Wangda Tan
13fa2d4e3e YARN-7394. Merge code paths for Reservation/Plan queues and Auto Created queues. (Suma Shivaprasad via wangda) 2017-11-06 21:38:24 -08:00
Haibo Chen
8f214dc4f8 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 2017-11-06 15:45:37 -08:00
Jian He
a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Billie Rinaldi
a127f7b0fb YARN-7423. Improve service client loggings. Contributed by Jian He 2017-11-06 13:30:20 -08:00
Billie Rinaldi
a6c4bd74b6 YARN-6394. Support specifying YARN related params in the service REST API. Contributed by Jian He 2017-11-06 13:30:20 -08:00
Jian He
a288b0accf YARN-7331. Change in few metrics in new YARN UI related to native-services. Contributed by Sunil G 2017-11-06 13:30:20 -08:00
Jian He
ba7ed7b66a YARN-7384. Remove apiserver cmd and merge service cmd into application cmd. Contributed by Billie Rinaldi 2017-11-06 13:30:19 -08:00
Billie Rinaldi
8ec9e72cc3 YARN-7393. RegistryDNS doesn't work in tcp channel. Contributed by Eric Yang 2017-11-06 13:30:19 -08:00
Billie Rinaldi
d2775fbcca YARN-7351. Fix high CPU usage issue in RegistryDNS. Contributed by Jian He 2017-11-06 13:30:19 -08:00
Billie Rinaldi
501be9b4be YARN-7326. Add recursion support and configure RegistryDNS to lookup upstream. Contributed by Eric Yang 2017-11-06 13:30:19 -08:00
Billie Rinaldi
68acd88dcb YARN-7323. Data structure update in service REST API. Contributed by Jian He 2017-11-06 13:30:18 -08:00
Jian He
3c30b1a97d Rebase to trunk 2017-11-06 13:30:18 -08:00
Jian He
9273480084 YARN-7198. Add jsvc support for RegistryDNS. Contributed by Billie Rinaldi 2017-11-06 13:30:18 -08:00
Jian He
b57144a172 YARN-7202. Add UT for api-server. Contributed by Eric Yang 2017-11-06 13:30:18 -08:00
Jian He
b8a7ef1b64 YARN-6744. Recover component information on YARN native services AM restart. Contributed by Billie Rinaldi 2017-11-06 13:30:18 -08:00
Jian He
c723021579 YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He
673c0db43c Revert "YARN-6626. Embed REST API service into RM. Contributed by Eric Yang" 
This reverts commit 63d1084e9781e0fee876916190b69f6242dd00e4.
 2017-11-06 13:30:17 -08:00
Jian He
9e677fa05c YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Billie Rinaldi
8851209788 YARN-7191. Improve yarn-service documentation. Contributed by Jian He 2017-11-06 13:30:17 -08:00
Jian He
c70e5df100 YARN-7201. Added an apache httpd example YARN service. Contributed by Billie Rinaldi 2017-11-06 13:30:16 -08:00
Billie Rinaldi
ce74e64363 YARN-7210. Some NPE fixes in Registry DNS. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Billie Rinaldi
37c9b7327d YARN-7186. Add examples in yarn-service. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Jian He
307d55b3e1 Rebased onto trunk - fix conflicts 2017-11-06 13:30:15 -08:00
Gour Saha
541b64c798 YARN-7165. Miscellaneous fixes in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:15 -08:00
Jian He
b9ca13e34b YARN-7161. Fix special chars in DNS documentation. Contributed by Gour Saha 2017-11-06 13:30:15 -08:00
Jian He
eed9e3e8f4 Rebase onto trunk - fix some issues. Contributed by Rohith Sharma K S 2017-11-06 13:30:14 -08:00
Jian He
e660e52a04 YARN-7126. Create introductory site documentation for YARN native services. Contributed by Gour Saha 2017-11-06 13:30:14 -08:00
Jian He
7dd4490a44 YARN-7073. Yarn native services rest API documentation. Contributed by Gour Saha 2017-11-06 13:30:14 -08:00
Jian He
bd96c4c235 Rebase onto latest trunk. minor conflicts 2017-11-06 13:30:13 -08:00
Jian He
59fe98cd50 YARN-7100. Added JsonIgnore annotation to Resource#getMemoryMB. Contributed by Eric Yang 2017-11-06 13:30:13 -08:00
Jian He
fecf22b2fd YARN-7113. Clean up packaging and dependencies for yarn-native-services. Contributed by Billie Rinaldi 2017-11-06 13:30:13 -08:00
Billie Rinaldi
40ab068eab YARN-7091. Rename application to service in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:13 -08:00
Gour Saha
2ba38ce8ec YARN-5244. Documentation required for DNS Server implementation. Contributed by Jon Maron 2017-11-06 13:30:12 -08:00
Billie Rinaldi
2191f5b533 YARN-7050. Post cleanup after YARN-6903, removal of org.apache.slider package. Contributed by Jian He 2017-11-06 13:30:12 -08:00
Billie Rinaldi
79bf5c084d YARN-7029. Add more UTs in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Billie Rinaldi
1888318c89 YARN-6903. Yarn-native-service framework core rewrite. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Jian He
27dcc67959 Update pom version to 3.0.0-beta1-SNAPSHOT 2017-11-06 13:30:10 -08:00
Jian He
8d335e59cf YARN-6804. [yarn-native-services changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-11-06 13:30:10 -08:00
Billie Rinaldi
14f3f3b42e YARN-6655. Fix compilation failure in TestServiceApiUtil due to changes in YARN-6716. 2017-11-06 13:30:10 -08:00
Jian He
59a9b4f438 YARN-6655. Validate yarn native services application submission side to ensure that the hostname should be less than 63 characters. Contributed by Billie Rinaldi 2017-11-06 13:30:09 -08:00
Jian He
e86d828548 YARN-6716. Native services support for specifying component start order. Contributed by Billie Rinaldi 2017-11-06 13:30:09 -08:00
Akira Ajisaka
b7e5d739ba YARN-6722. Bumping up pom file hadoop version. Contributed by Panagiotis Garefalakis. 2017-11-06 13:30:09 -08:00
Billie Rinaldi
a0b9e8a630 YARN-6617. Services API delete call first attempt usually fails. Contributed by Jian He 2017-11-06 13:30:09 -08:00
Jian He
a0574e7f4f YARN-6613. Update json validation for new native services providers. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Jian He
9085cd5c6e YARN-6160. Create an agent-less docker-less provider in the native services framework. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Gour Saha
a22256bd40 YARN-6544. Add Null check RegistryDNS service while parsing registry records. Contributed by Karam Singh 2017-11-06 13:30:08 -08:00
Billie Rinaldi
02d68c4bcf YARN-6588. Add native-service AM log4j properties in classpath. Contributed by Jian He 2017-11-06 13:30:08 -08:00
Jian He
a041373dd4 YARN-6533. Race condition in writing service record to registry in yarn native services. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Billie Rinaldi
ce05c6e981 YARN-6545. Followup fix for YARN-6405. Contributed by Jian He 2017-11-06 13:30:07 -08:00
Sunil G
399525c2e0 YARN-6398. Support to add native-service specific details in new YARN UI. Contributed by Akhil PB. 2017-11-06 13:30:07 -08:00
Sunil G
ba812bf67f YARN-6446. addendum patch. Contributed by Rohith Sharma K S. 2017-11-06 13:30:07 -08:00
Jian He
a8a273b064 YARN-6446. Revisit ATSv2 integration to ensure all required information is published. Contributed by Rohith Sharma K S 2017-11-06 13:30:07 -08:00
Sunil G
08cdf231a0 YARN-6419. Support to launch new native-service from new YARN UI. Contributed by Akhil PB. 2017-11-06 13:30:06 -08:00
Billie Rinaldi
845764da14 YARN-6405. Improve configuring services through REST API. Contributed by Jian He 2017-11-06 13:28:42 -08:00
Jian He
384ee13eec YARN-6335. Port slider's groovy unit tests to yarn native services. Contributed by Billie Rinaldi 2017-11-06 13:28:42 -08:00
Jian He
c3ac67e3e1 Rebase to latest trunk, fix conflicts and pom.xml 2017-11-06 13:28:41 -08:00
Jian He
fd9d0d4ead YARN-6400. Remove some unneeded code after YARN-6255. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Jian He
f4216b7bba YARN-6395. Integrate service app master to write data into ATSv2. Contributed by Rohith Sharma K S 2017-11-06 13:28:41 -08:00
Jian He
c31cd981eb YARN-6255. Refactor yarn-native-services framework. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Billie Rinaldi
024e90a508 YARN-6192. Move yarn-native-service API records into hadoop-yarn-slider-core module. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Gour Saha
0ed8d4ebe6 YARN-6205. Default lifetime for native services app is invalid. Contributed by Billie Rinaldi 2017-11-06 13:28:40 -08:00
Gour Saha
500695d726 YARN-6185. Apply SLIDER-1199 to yarn native services for blacklisting nodes. Contributed by Billie Rinaldi 2017-11-06 13:28:40 -08:00
Gour Saha
8967a1b812 YARN-6186 Handle InvalidResourceRequestException in native services AM onError 2017-11-06 13:28:40 -08:00
Billie Rinaldi
e34ca6b02d YARN-6173. Add artifact info and privileged container details to the container info in API GET response. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Jian He
68940ab8f6 Updated pom to point to 3.0.0-alpha3-SNAPSHOT 2017-11-06 13:28:40 -08:00
Billie Rinaldi
d943eb4c15 YARN-6132. SliderClient bondToCluster should call findInstance with live states filter. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Billie Rinaldi
7edc154e0e YARN-6115. Few additional paths in Slider client still uses get all Applications without tags/states filter. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Gour Saha
a5e20f0fc1 YARN-5993. Allow native services quicklinks to be exported for each component. Contributed by Billie Rinaldi 2017-11-06 13:28:39 -08:00
Varun Vasudev
7d685f2fb3 YARN-4757. Add the ability to split reverse zone subnets. Contributed by Shane Kumpf. 
(cherry picked from commit 9bff70f1316d889c3109c68dd308c8f456229bf8)
 2017-11-06 13:28:39 -08:00
Jian He
f934f62fc7 YARN-5218. Initial core change for DNS for YARN. Contributed by Jonathan Maron 2017-11-06 13:28:39 -08:00
Billie Rinaldi
74fff4086e YARN-6014. Followup fix for slider core module findbugs. Contributed by Jian He 2017-11-06 13:28:39 -08:00
Billie Rinaldi
1ea43c443e YARN-6010. Fix findbugs, site warnings in yarn-services-api module. Contributed by Jian He 2017-11-06 13:28:38 -08:00
Jian He
0b00c8e56b YARN-5968. Fix slider core module javadocs. Contributed by Billie Rinaldi 2017-11-06 13:28:38 -08:00
Billie Rinaldi
db96e8aa21 YARN-5967. Fix slider core module findbugs warnings. Contributed by Jian He 2017-11-06 13:28:38 -08:00
Gour Saha
c11f4b3c21 YARN-5996. Native services AM kills app on AMRMClientAsync onError call. Contributed by Billie Rinaldi 2017-11-06 13:28:38 -08:00
Gour Saha
2e1fa61253 YARN-5740. Add a new field in Slider status output - lifetime (remaining). Contributed by Jian He 2017-11-06 13:28:37 -08:00
Billie Rinaldi
2cea59dc54 YARN-5975. Remove the agent - slider AM ssl related code. Contributed by Jian He 2017-11-06 13:28:37 -08:00
Jian He
efe7d44ead YARN-5961. Generate native services protobuf classes during build. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha
e4f0263e51 YARN-5958. Fix ASF license warnings for slider core module. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha
1f44e6a5f4 YARN-5944. Native services AM should remain up if RM is down. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha
6143aa135b YARN-5769. Integrate update app lifetime using feature implemented in YARN-5611. Contributed by Jian He 2017-11-06 13:28:37 -08:00
Gour Saha
bd0e9f91b6 YARN-5941. Slider handles "per.component" for multiple components incorrectly. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha
4f43ebdb9f YARN-5943. Write native services container stderr file to log directory. Contributed by Billie Rinaldi 2017-11-06 13:28:36 -08:00
Billie Rinaldi
7b8fd3abd6 YARN-5883 Avoid or eliminate expensive YARN get all applications call. Contributed by Gour Saha 2017-11-06 13:28:36 -08:00
Jian He
234dba84a1 YARN-5909. Remove agent related code in slider AM. Contributed by Jian He 2017-11-06 13:28:36 -08:00
Gour Saha
2c4e0f33e9 YARN-5808. Add gc log options to the yarn daemon script when starting services-api. Contributed by Billie Rinaldi 2017-11-06 13:28:35 -08:00
Gour Saha
d1ee95981e YARN-5828. Native services client errors out when config formats are uppercase. Contributed by Billie Rinaldi 2017-11-06 13:28:35 -08:00
Jian He
7757f6fe4a YARN-5812. Exception during GET call - "Failed to retrieve application: null". Contributed by Gour Saha 2017-11-06 13:28:35 -08:00
Gour Saha
25411a1742 YARN-5813. Slider should not try to set a negative lifetime timeout value. Contributed by Jian He 2017-11-06 13:28:35 -08:00
Jian He
3741e5518f YARN-5796. Convert enums values in service code to upper case and special handling of an error. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha
09e4b9e897 YARN-5690. Integrate native services modules into maven build. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Billie Rinaldi
ef5a3628c2 YARN-5770. Performance improvement of native-services REST API service. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha
1d7a42e795 YARN-5680. Add 2 new fields in Slider status output - image-name and is-privileged-container. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Billie Rinaldi
773c565ac5 YARN-5775. Convert enums in swagger definition to uppercase. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha
6bdfce3ac7 YARN-5778. Add .keep file for yarn native services AM web app. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Jian He
f206a43954 YARN-5701. Fix issues in yarn native services apps-of-apps. Contributed by Billie Rinaldi 2017-11-06 13:28:33 -08:00
Gour Saha
4ec1cbe86d YARN-5735. Make the service REST API use the app timeout feature YARN-4205. Contributed by Jian He 2017-11-06 13:28:33 -08:00
Jian He
7da243ebe0 YARN-5729. Bug fixes for the service Rest API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He
c0120e51b9 YARN-5689. Update native services REST API to use agentless docker provider. Contributed by Billie Rinaldi & Gour Saha 2017-11-06 13:28:33 -08:00
Jian He
f66ecfecf6 Addendum patch for YARN-5610. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He
295b87d0fb YARN-5675. Swagger definition for YARN service API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He
18bf0dce28 YARN-5610. Initial code for native services REST API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He
9dc46aa379 YARN-5623. Apply SLIDER-1166 to yarn-native-services branch. Contributed by Gour Saha 2017-11-06 13:28:32 -08:00
Jian He
bce06ed1af YARN-5505. Create an agent-less docker provider in the native-services framework. Contributed by Billie Rinaldi 2017-11-06 13:28:32 -08:00
Jian He
3279baecb9 YARN-5538. Apply SLIDER-875 to yarn-native-services. Contributed by Billie Rinaldi 2017-11-06 13:28:32 -08:00
Jian He
02e2740bdf YARN-5513. Move Java only tests from slider develop to yarn-native-services. Contributed by Gour Saha 2017-11-06 13:28:32 -08:00
Jian He
54ec9e2440 Modify pom file for slider 2017-11-06 13:28:32 -08:00
Jian He
804bb82d1b Rename org.apache.slider.core.build to org.apache.slider.core.buildutils 2017-11-06 13:28:32 -08:00
Jian He
f47df51791 YARN-5461. Initial code ported from slider-core module. (jianhe) 2017-11-06 13:28:31 -08:00
bibinchundatt
dcd99c4b9a Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 2017-11-06 22:39:10 +05:30
Sunil G
c41728486b YARN-7392. Render cluster information on new YARN web ui. Contributed by Vasudevan Skm. 2017-11-03 11:45:50 +05:30
Inigo Goiri
6fc09beac4 YARN-7434. Router getApps REST invocation fails with multiple RMs. Contributed by Inigo Goiri. 2017-11-02 21:29:53 -07:00
Sunil G
1700adc6f7 YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. Contributed by Wangda Tan. 2017-11-03 07:25:29 +05:30
Subru Krishnan
ad0fff2b41 YARN-7432. Fix DominantResourceFairnessPolicy serializable findbugs issues. (Daniel Templeton via Subru). 2017-11-02 17:39:23 -07:00
Eric Payne
e6ec02001f YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 2017-11-02 12:37:33 -05:00
Wangda Tan
04c604cf1d YARN-7364. Queue dash board in new YARN UI has incorrect values. (Sunil G via wangda) 
Change-Id: Ia5336f407147d2985a61b5f9f1e69b5b35f398a4
(cherry picked from commit 752f000e0f521ca7be173d6ca338da4585284fd0)
 2017-11-02 09:51:28 -07:00
Jason Lowe
d00b6f7c1f YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 2017-11-02 09:37:17 -05:00
Rohith Sharma K S
940ffe3f9c addendum patch for YARN-7289. 2017-11-02 13:55:19 +05:30
Jian He
0cc98ae0ec YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2017-11-01 17:00:32 -07:00
Eric Yang
7a49ddfdde YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:39:56 -04:00
Junping Du
d57dba9942 YARN-7400. Incorrect log preview displayed in jobhistory server ui. Contributed by Xuan Gong. 2017-11-01 14:48:16 -07:00
Inigo Goiri
70f1a9470c YARN-7276 addendum to add timeline service depencies. Contributed by Inigo Goiri. 2017-11-01 13:26:37 -07:00
Subru Krishnan
ed24da3dd7 YARN-6413. FileSystem based Yarn Registry implementation. (Ellen Hui via Subru). 2017-10-31 12:05:43 -07:00
Akira Ajisaka
c02d2ba50d
YARN-7379. Moving logging APIs over to slf4j in hadoop-yarn-client. Contributed by Yeliang Cang. 2017-10-31 17:09:45 +09:00
Akira Ajisaka
785f1b0d11
YARN-7407. Moving logging APIs over to slf4j in hadoop-yarn-applications. Contributed by Yeliang Cang. 2017-10-31 16:36:02 +09:00
Junping Du
a8083aaa6f HADOOP-14990. Clean up jdiff xml files added for 2.8.2 release. 2017-10-30 15:16:51 -07:00
Daniel Templeton
d64736d589 YARN-7336. Unsafe cast from long to int Resource.hashCode() method 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-10-30 12:41:28 -07:00
Daniel Templeton
9a7e810838 YARN-6927. Add support for individual resource types requests in MapReduce 
(Contributed by Gergo Repas via Daniel Templeton)
 2017-10-30 11:04:22 -07:00
Daniel Templeton
9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00
Yufei Gu
d4811c8cfa YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 2017-10-29 16:44:16 -07:00
Sunil G
9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Daniel Templeton
e62bbbca7a YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() 2017-10-28 09:13:13 -07:00
Arun Suresh
9c5c68745e YARN-7299. Fix TestDistributedScheduler. (asuresh) 2017-10-27 23:08:18 -07:00
Arun Suresh
24f8c5cce3 YARN-7178. Add documentation for Container Update API. (asuresh) 2017-10-27 22:56:41 -07:00
Inigo Goiri
8be5707067 YARN-7276. Federation Router Web Service fixes. Contributed by Inigo Goiri. 2017-10-27 16:46:05 -07:00
Jason Lowe
665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Rohith Sharma K S
5c799ecf09 YARN-7289. Application lifetime does not work with FairScheduler. Contributed by Miklos Szegedi. 2017-10-27 22:46:38 +05:30
Sunil G
792388e1c0 YARN-7375. Possible NPE in RMWebapp when HA is enabled and the active RM fails. Contributed by Chandni Singh. 2017-10-27 20:53:57 +05:30
Wangda Tan
36e158ae98 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 
Change-Id: I14c5ea7252b7c17e86ab38f692b5f9d43196dbe0
 2017-10-26 20:15:19 -07:00
Junping Du
a25b5aa0cf Add several jdiff xml files for 2.8.2 release. 2017-10-26 19:30:48 -07:00
Robert Kanter
b1de78619f YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 2017-10-26 17:47:32 -07:00
Robert Kanter
088ffee716 YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ Addendum (misha@cloudera.com via rkanter) 2017-10-26 15:50:14 -07:00
Subru Krishnan
25932da6d1 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 2017-10-26 12:10:14 -07:00
Robert Kanter
2da654e34a YARN-7358. TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler (haibochen via rkanter) 2017-10-26 10:25:10 -07:00
Subru Krishnan
3fae675383 YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru). 2017-10-25 15:07:50 -07:00
Haibo Chen
d7f3737f3b YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 2017-10-24 22:17:56 -07:00
Robert Kanter
03af442e76 YARN-7385. TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE (yufeigu via rkanter) 2017-10-24 13:36:50 -07:00
Carlo Curino
1c5c2b5dde YARN-7339. LocalityMulticastAMRMProxyPolicy should handle cancel request properly. (Botong Huang via curino) 2017-10-24 10:39:04 -07:00
Robert Kanter
025c656572 YARN-7382. NoSuchElementException in FairScheduler after failover causes RM crash (rkanter) 2017-10-24 10:21:44 -07:00
Daniel Templeton
679f99b145 YARN-6984. DominantResourceCalculator.isAnyMajorResourceZero() should test all resources 
(Contributed by Sunil G via Daniel Templeton)
 2017-10-24 10:08:26 -07:00
Robert Kanter
5da295a34e YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ (misha@cloudera.com via rkanter) 2017-10-23 17:56:56 -07:00
Daniel Templeton
9e77dc2bd1 YARN-7357. Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-10-23 13:51:19 -07:00
Eric Payne
921338cd86 YARN-4163: Audit getQueueInfo and getApplications calls 2017-10-23 11:43:41 -05:00
Haibo Chen
480187aebb YARN-7372. TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic is flaky. 2017-10-20 14:24:17 -07:00
Yufei Gu
6b7c87c945 YARN-7355. TestDistributedShell should be scheduler agnostic. (Contributed by Haibo) 2017-10-20 11:15:20 -07:00
Yufei Gu
0799fde35e YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 2017-10-20 10:00:13 -07:00
Eric Yang
b61144a93d YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-20 12:02:06 -04:00
Yufei Gu
1f4cdf1068 YARN-4090. Make Collections.sort() more efficient by caching resource usage. (Contributed by Yufei Gu, Shilong Zhang and Xianyin Xin) 2017-10-20 01:32:20 -07:00
Wangda Tan
4afd308b62 YARN-7170. Improve bower dependencies for YARN UI v2. (Sunil G via wangda) 2017-10-19 21:58:40 -07:00
Yufei Gu
7b4b018780 YARN-7359. TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic. (Contributed by Haibo Chen) 2017-10-19 16:51:47 -07:00
Yufei Gu
cbd2b73ef8 YARN-7294. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair Scheduler. (Contributed by Miklos Szegedi) 2017-10-19 16:39:25 -07:00
Wangda Tan
c1b08ba720 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2017-10-19 14:45:44 -07:00
Wangda Tan
298b174f66 YARN-7338. Support same origin policy for cross site scripting prevention. (Sunil G via wangda) 2017-10-19 14:44:42 -07:00
Subru Krishnan
75323394fb YARN-7311. Fix TestRMWebServicesReservation parametrization for fair scheduler. (Yufei Gu via Subru). 2017-10-17 12:38:06 -07:00
Haibo Chen
acabc657ff YARN-7341. TestRouterWebServiceUtil#testMergeMetrics is flakey. (Robert Kanter via Haibo Chen) 2017-10-17 10:15:53 -07:00
Robert Kanter
8a61525928 YARN-7308. TestApplicationACLs fails with FairScheduler (rkanter) 2017-10-16 15:34:32 -07:00
Nathan Roberts
4540ffd15f YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe. 2017-10-16 17:00:38 -05:00
Junping Du
1fcbe7cf5f YARN-7124. LogAggregationTFileController deletes/renames while file is open. Contributed by Jason Lowe. 2017-10-16 13:57:03 -07:00
Arun Suresh
a50be1b8f4 YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh) 2017-10-16 13:12:15 -07:00
Wangda Tan
87ea1dff9c Addendum fix for: YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. (Wangda Tan) 
Change-Id: If1fe4a62e07b25e6f1b8ba803495da87e2cb2df6
 2017-10-14 10:41:58 -07:00
Haibo Chen
3fb4718886 YARN-7310. TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler. (Robert Kanter via Haibo Chen) 2017-10-13 15:49:21 -07:00
Yufei Gu
7a27c2c367 YARN-7270. Fix unsafe casting from long to int for class Resource and 
its sub-classes. (Yufei)
 2017-10-13 12:41:59 -07:00
Rohith Sharma K S
f4fb6695a3 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-13 15:17:05 +05:30
Carlo Curino
13fcfb3d46 YARN-7317. Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy. (contributed by Botong Huang via curino) 2017-10-12 10:38:58 -07:00
Wangda Tan
8bcc49e677 YARN-7205. Log improvements for the ResourceUtils. (Sunil G via wangda) 
Change-Id: I0f5b7a7f68ec5d3e1d52211f83fdd089bc0bfd37
 2017-10-11 15:25:28 -07:00
Sunil G
fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Jason Lowe
ebb34c7053 YARN-7082. TestContainerManagerSecurity failing in trunk. Contributed by Akira Ajisaka 2017-10-11 10:31:02 -05:00
Robert Kanter
ec8bf9e48a YARN-7309. TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler (rkanter) 2017-10-10 10:18:42 -07:00
Sunil G
41351b05f3 YARN-7296. convertToProtoFormat(Resource r) is not setting for all resource types. Contributed by lovekesh bansal. 2017-10-10 12:02:18 +05:30
Wangda Tan
5cfbad68c0 YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) 
Change-Id: I959c8531bd48231956068cb7d2312f3235549f9c
(cherry picked from commit 8536db5a7549fa44d9f9e444b41bd5978b5c6ae1)
 2017-10-09 20:36:57 -07:00
Jonathan Hung
1d36b53ab6 YARN-7252. Removing queue then failing over results in exception 2017-10-09 11:11:23 -07:00
Jonathan Hung
09c5dfe937 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:11:23 -07:00
Jonathan Hung
a3075de9ab YARN-7238. Documentation for API based scheduler configuration management 2017-10-09 11:11:23 -07:00
Jonathan Hung
d6622daaa3 YARN-7046. Add closing logic to configuration store 2017-10-09 11:11:23 -07:00
Wangda Tan
ff39c0de20 YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:11:23 -07:00
Xuan
4d8abd84f4 YARN-7024: Fix issues on recovery in LevelDB store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
4c8b208adb YARN-6322: Disable queue refresh when configuration mutation is enabled. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
74ba6ffa0b YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
04f9e80bb2 YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:11:23 -07:00
Xuan
916bdbd6be YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:11:23 -07:00
Xuan
e566fd8b58 YARN-6575. Support global configuration mutation in MutableConfProvider. (Jonathan Hung via Xuan Gong) 2017-10-09 11:11:23 -07:00
Wangda Tan
a4e6253046 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:11:23 -07:00
Wangda Tan
d8a5136b1f YARN-5952. Create REST API for changing YARN scheduler configurations. (Jonathan Hung via wangda) 2017-10-09 11:11:23 -07:00
Jonathan Hung
ef686635c1 YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:11:23 -07:00
Xuan
e3579a8c3b YARN-5946: Create YarnConfigurationStore interface and 
InMemoryConfigurationStore class. Contributed by Jonathan Hung
 2017-10-09 11:11:23 -07:00
Jonathan Hung
ef59cbe08a YARN-5951. Changes to allow CapacityScheduler to use configuration store 2017-10-09 11:11:22 -07:00
Daniel Templeton
6d6ca4c923 YARN-2612 addendum: fixed javadoc error. 2017-10-09 14:12:13 +02:00
Sunil G
8137254a1f YARN-7245. Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user. Contributed by Eric Payne. 2017-10-09 14:39:41 +05:30
Rohith Sharma K S
6fe6a72352 Revert "YARN-7180. Remove class ResourceType. Contributed by Sunil G." 
This reverts commit 6bd81ab054.
 2017-10-09 13:21:13 +05:30
Rohith Sharma K S
6bd81ab054 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-09 10:20:40 +05:30
Yufei Gu
72d22b753a YARN-7207. Cache the RM proxy server address. (Yufei Gu) 2017-10-06 09:49:11 -07:00
Yufei Gu
49ae538164 YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage(addendum). (Yufei Gu) 2017-10-06 00:08:25 -07:00
Yufei Gu
99292adcef YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-05 23:51:58 -07:00
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Naganarasimha
d0046bb5c6 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 2017-10-06 00:07:53 +05:30
Chris Trezzo
7e76f85bc6 YARN-2960. Add documentation for the YARN shared cache. 2017-10-05 10:38:41 -07:00
Arun Suresh
b733348dde YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 2017-10-05 09:58:04 -07:00
Sunil G
f702c95754 YARN-6182. Fix alignment issues and missing information in new YARN UI's Queue page. Contributed by Akhil PB. 2017-10-05 11:34:44 +05:30
Sunil G
592bf2d550 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton. 2017-10-05 10:52:00 +05:30
Subru Krishnan
e6e614e380 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-10-04 19:28:27 -07:00
Akira Ajisaka
2df1b2ac05
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails. 2017-10-05 06:07:32 +09:00
Akira Ajisaka
acf5b880d8
YARN-6975. Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router. Contributed by Yeliang Cang. 2017-10-04 14:53:41 +09:00
Ray Chiang
67e12169b0 YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang) 2017-10-03 15:06:07 -07:00
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Akira Ajisaka
4a87773718
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka. 2017-10-04 06:06:39 +09:00
Allen Wittenauer
4d5dd75b60
HADOOP-14908. CrossOriginFilter should trigger regex on more input (Johannes Alberti via aw) 2017-10-03 10:58:28 -07:00
Jian He
4111e6c781 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan 2017-10-03 09:54:38 -07:00
Sidharta S
7eb8499996 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-02 19:04:49 -07:00
Subru Krishnan
d4d2fd1acd YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru). 2017-10-02 18:14:44 -07:00
Wangda Tan
280080fad0 YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda) 
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
 2017-10-02 15:30:22 -07:00
Carlo Curino
ceca9694f9 YARN-3661. Basic Federation UI. (Contributed by Inigo Goiri via curino) 2017-10-02 13:03:32 -07:00
Yufei Gu
66c417167a YARN-6333. Improve doc for minSharePreemptionTimeout, fairSharePreemptionTimeout and fairSharePreemptionThreshold. (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-29 18:07:33 -07:00
Wangda Tan
ec2ae3060a YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda) 
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
 2017-09-29 16:42:56 -07:00
Wangda Tan
febeead5f9 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
 2017-09-29 16:39:46 -07:00
Eric Payne
7f6118f918 YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe. 2017-09-29 10:33:57 -05:00
Wangda Tan
d3b1c63195 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
 2017-09-28 16:41:09 -07:00
Chris Trezzo
c114da5e64 YARN-7250. Update Shared cache client api to use URLs. 2017-09-28 15:28:06 -07:00
Subru Krishnan
ca669f9f8b YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 2017-09-28 13:04:03 -07:00
Jason Lowe
85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Xuan
28c4957fcc YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad 
'start' value. Contributed by Jason Lowe
 2017-09-27 15:46:18 -07:00
Chris Trezzo
c87db8d154 YARN-7253. Shared Cache Manager daemon command listed as admin subcmd in yarn script. 2017-09-27 11:32:09 -07:00
Sunil G
8facf1f976 YARN-6871. Add additional deSelects params in RMWebServices#getAppReport. Contributed by Tanuj Nayak. 2017-09-27 14:37:32 +05:30
Jason Lowe
14fec04ed3 Revert "YARN-6570. No logs were found for running application, running" 
This reverts commit 7c732924a8.
 2017-09-26 10:38:03 -05:00
Rohith Sharma K S
06e5a7b5cf YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R. 2017-09-26 14:34:09 +05:30
Arun Suresh
df800f6cf3 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh) 2017-09-25 16:02:55 -07:00
Akira Ajisaka
e928ee583c
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao. 2017-09-26 02:03:30 +09:00
Daniel Templeton
164a063e74 YARN-6878. TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-09-22 23:53:32 +02:00
Junping Du
4002bf0a9e YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi. 2017-09-22 13:58:09 -07:00
Jason Lowe
bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Junping Du
e5e1851d80 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:01:16 -07:00
Yufei Gu
a92ef030a2 YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 2017-09-21 12:31:07 -07:00
Wangda Tan
263e2c692a YARN-6991. "Kill application" button does not show error if other user tries to kill the application for secure cluster. (Suma Shivaprasad via wangda) 
Change-Id: I7e7894b24609709f89064ee5882f055dbb09080b
 2017-09-21 12:00:53 -07:00
Jason Lowe
10d7493587 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 2017-09-21 08:38:08 -05:00
Sangjin Lee
8b336632ac YARN-6771. Use classloader inside configuration class to make new 
classes. Contributed by Jongyoul Lee.
 2017-09-20 22:22:43 -07:00
Yufei Gu
a9019e1fb7 YARN-6499. Remove the doc about Schedulable#redistributeShare(). (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-19 18:28:31 -07:00
Junping Du
647b7527a9 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 2017-09-19 18:31:15 -07:00
Arun Suresh
56ef5279c1 YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 2017-09-18 15:49:31 -07:00
Andrew Wang
3cf3540f19 HADOOP-14835. mvn site build throws SAX errors. Contributed by Andrew Wang and Sean Mackrory. 2017-09-18 15:13:42 -07:00
Xuan
7c732924a8 YARN-6570. No logs were found for running application, running 
container. Contributed by Junping Du
 2017-09-18 14:04:05 -07:00
Arun Suresh
29dd55153e YARN-7199. Fix TestAMRMClientContainerRequest.testOpportunisticAndGuaranteedRequests. (Botong Huang via asuresh) 2017-09-18 11:26:44 -07:00
Jason Lowe
a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Wangda Tan
e81596d06d YARN-7172. ResourceCalculator.fitsIn() should not take a cluster resource parameter. (Sen Zhao via wangda) 
Change-Id: Icc3670c9381ce7591ca69ec12da5aa52d3612d34
 2017-09-17 21:20:43 -07:00
Wangda Tan
38c14ef8d8 YARN-7149. Cross-queue preemption sometimes starves an underserved queue. (Eric Payne via wangda) 
Change-Id: Ib269991dbebce160378e8372ee6d24849c4a5ed6
(cherry picked from commit 3dfa937a1fadfc62947755872515f549b3b15e6a)
 2017-09-15 21:29:39 -07:00
Wangda Tan
958e8c0e25 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
 2017-09-15 21:29:31 -07:00
Junping Du
1a84c24b0c YARN-7174. Add retry logic in LogsCLI when fetch running application logs. Contributed by Xuan Gong. 2017-09-15 15:33:24 -07:00
Yufei Gu
09b476e6da YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 2017-09-14 11:23:37 -07:00
Arun Suresh
66ca0a6540 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00