big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,890 Commits 2 Branches 2 Tags 264 MiB
985161f6ec
Commit Graph

944 Commits

Author SHA1 Message Date
Shane Kumpf
dffb7bfe6c YARN-8638. Allow linux container runtimes to be pluggable. Contributed by Craig Condit 2018-09-05 06:47:54 -06:00
Shane Kumpf
73625168c0 YARN-8642. Add support for tmpfs mounts with the Docker runtime. Contributed by Craig Condit 2018-08-29 07:08:37 -06:00
Giovanni Matteo Fumarola
e0f6ffdbad YARN-8581. [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy. Contributed by Botong Huang. 2018-08-20 14:33:16 -07:00
Rohith Sharma K S
1697a02306 YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli. 2018-08-17 11:12:10 +05:30
Weiwei Yang
8dc07b4723 YARN-8614. Fix few annotation typos in YarnConfiguration. Contributed by Sen Zhao. 2018-08-15 12:15:54 +08:00
Sunil G
12a095a496 YARN-7159. Normalize unit of resource objects in RM to avoid unit conversion in critical path. Contributed by Manikandan R. 2018-08-02 22:29:21 +05:30
Jonathan Hung
3e06a5dcea YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:44:18 -07:00
Robert Kanter
2cccf4061c YARN-8517. getContainer and getContainers ResourceManager REST API methods are not documented (snemeth via rkanter) 2018-07-27 14:35:03 -07:00
bibinchundatt
8461278833 YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt. 2018-07-24 13:09:17 +05:30
Haibo Chen
f354f47f99 YARN-6995. Improve use of ResourceNotFoundException in resource types code. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-19 15:35:05 -07:00
Wangda Tan
238ffff999 YARN-8524. Single parameter Resource / LightWeightResource constructor looks confusing. (Szilard Nemeth via wangda) 
Change-Id: I4ae97548b5b8d76a6bcebb2d3d70bf8e0be3c125
 2018-07-16 10:58:00 -07:00
Rohith Sharma K S
ba68320449 YARN-8302. ATS v2 should handle HBase connection issue properly. Contributed by Billie Rinaldi. 2018-07-06 15:19:01 -07:00
Haibo Chen
9edc74f64a YARN-7556. Fair scheduler configuration should allow resource types in the minResources and maxResources properties. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-06 11:03:48 -07:00
Anu Engineer
39ad98903a Revert "Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/hadoop into trunk" 
This reverts commit c163d1797a, reversing
changes made to 0d9804dcef.
 2018-07-05 12:22:18 -07:00
Haibo Chen
1726247024 YARN-7556. Fair scheduler configuration should allow resource types in the minResources and maxResources properties. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-05 10:42:39 -07:00
Inigo Goiri
99948565cb YARN-8412. Move ResourceRequest.clone logic everywhere into a proper API. Contributed by Botong Huang. 2018-06-21 18:24:10 -07:00
Miklos Szegedi
55432b0981 YARN-8440. Typo in YarnConfiguration javadoc: "Miniumum request grant-able..". Contributed by Szilard Nemeth. 2018-06-20 11:36:12 -07:00
Haibo Chen
24a89825f0 YARN-6931. Make the aggregation interval in AppLevelTimelineCollector configurable. (Abhishek Modi via Haibo Chen) 2018-06-12 10:03:34 -07:00
Akira Ajisaka
652bcbb3e4
YARN-8363. Upgrade commons-lang version to 3.7 in hadoop-yarn-project. Contributed by Takanobu Asanuma. 2018-06-13 00:35:37 +09:00
Shane Kumpf
1388de18ad YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:30:39 -06:00
Rohith Sharma K S
c05b5d424b YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 2018-05-24 14:19:46 +05:30
Haibo Chen
d996479954 YARN-4599. Set OOM control for memory cgroups. (Miklos Szegedi via Haibo Chen) 2018-05-23 16:35:37 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
328f0847e3 YARN-8128. Documenting the per-node per-app file limit in YARN log aggregation. Contributed by Xuan Gong. 2018-05-17 12:47:10 -07:00
Haibo Chen
e3b7d7ac16 YARN-7933. [atsv2 read acls] Add TimelineWriter#writeDomain. (Rohith Sharma K S via Haibo Chen) 2018-05-16 13:00:01 -07:00
Jason Lowe
6c8e51ca7e YARN-7654. Support ENTRY_POINT for docker container. Contributed by Eric Yang 2018-05-11 18:56:05 -05:00
Robert Kanter
c8b53c4364 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 2018-05-10 09:31:59 -07:00
Inigo Goiri
e6a80e476d YARN-8151. Yarn RM Epoch should wrap around. Contributed by Young Chen. 2018-05-02 17:23:17 -07:00
Konstantinos Karanasos
28e2244390 YARN-8111. Simplify PlacementConstraints API by removing allocationTagToIntraApp. Contributed by Weiwei Yang. 2018-04-20 12:24:48 -07:00
Haibo Chen
bb92bfb4ef YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 11:15:20 -07:00
Haibo Chen
86dfb70875 Revert "YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)" 
This reverts commit 0c14b6f0d8.
 2018-04-17 06:04:38 -07:00
Haibo Chen
0c14b6f0d8 YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 05:59:03 -07:00
Rohith Sharma K S
669eb7bdea YARN-8156. Increase the default value of yarn.timeline-service.app-collector.linger-period.ms. Contributed by Charan Hebri. 2018-04-14 10:31:28 +05:30
Eric Yang
c467f311d0 YARN-7973. Added ContainerRelaunch feature for Docker containers. 
Contributed by Shane Kumpf
 2018-04-10 19:25:00 -04:00
Jason Lowe
907919d28c YARN-7667. Docker Stop grace period should be configurable. Contributed by Eric Badger 2018-04-09 17:19:21 -05:00
Wangda Tan
00ebec89f1 YARN-1151. Ability to configure auxiliary services from HDFS-based JAR files. (Xuan Gong via wangda) 
Change-Id: Ied37ff11e507fc86847753ba79486652c8fadfe9
 2018-04-06 21:25:57 -07:00
Wangda Tan
d4e63ccca0 YARN-8048. Support auto-spawning of admin configured services during bootstrap of RM (Rohith Sharma K S via wangda) 
Change-Id: I2d8d61ccad55e1118009294d7e17822df3cd0fd5
 2018-04-06 21:24:58 -07:00
Haibo Chen
f8b8bd53c4 YARN-6936. [Atsv2] Retrospect storing entities into sub application table from client perspective. (Rohith Sharma K S via Haibo Chen) 2018-04-05 10:23:42 -07:00
Vrushali C
345e7624d5 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 2018-04-04 15:08:03 -07:00
Konstantinos Karanasos
7853ec8d2f YARN-8013. Support application tags when defining application namespaces for placement constraints. Contributed by Weiwei Yang. 2018-04-04 10:51:58 -07:00
Wangda Tan
a0bde7d525 YARN-7142. Support placement policy in yarn native services. (Gour Saha via wangda) 
Change-Id: I166c67a7a34430627c17365f60bac75b6da1b434
 2018-04-02 07:26:01 -07:00
Weiwei Yang
1606dad133 YARN-7497. Add file system based scheduler configuration store. Contributed by Jiandan Yang. 2018-03-30 21:41:33 +08:00
Subru Krishnan
09999d7e01 YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 2018-03-28 11:33:19 -07:00
Subru Krishnan
725b10e3ae Revert "YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru)." 
This reverts commit 2a2ef15caf as smart-apply-patch script didn't pick the latest patch.
 2018-03-28 11:26:50 -07:00
Subru Krishnan
2a2ef15caf YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 2018-03-27 17:39:46 -07:00
Wangda Tan
a90471b3e6 YARN-8016. Refine PlacementRule interface and add a app-name queue mapping rule as an example. (Zian Chen via wangda) 
Change-Id: I35caf1480e0f76f5f3a53528af09312e39414bbb
 2018-03-23 16:43:40 -07:00
Wangda Tan
a08921ca6c YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (Weiwei Yang via wangda) 
Change-Id: I63b4e4192a95bf7ded98c54e46a2871c72869700
 2018-03-19 11:04:27 -07:00
Wangda Tan
4bf622043f YARN-7952. RM should be able to recover log aggregation status after restart/fail-over. (Xuan Gong via wangda) 
Change-Id: I725c9afe64831eda0aa6b0bebdbc79d2dd165707
 2018-03-15 13:26:45 -07:00
Wangda Tan
a5b27b3c67 YARN-5015. Support sliding window retry capability for container restart. (Chandni Singh via wangda) 
Change-Id: I07addd3e4ba8d98456ee2ff1d5c540a38fe61dea
 2018-03-13 17:55:17 -07:00
Miklos Szegedi
a82d4a2e3a YARN-5764. NUMA awareness support for launching containers. Contributed by Devaraj K. 2018-03-13 12:36:57 -07:00
Arun Suresh
1054b48c27 YARN-7972. Support inter-app placement constraints for allocation tags by application ID. (Weiwei Yang via asuresh) 2018-03-05 11:24:17 -08:00
Weiwei Yang
e8c5be63f0 YARN-7995. Remove unnecessary boxings and unboxings from PlacementConstraintParser.java. Contributed by Sen Zhao. 2018-03-05 11:49:43 +08:00
Konstantinos Karanasos
e85188101c YARN-7921. Transform a PlacementConstraint to a string expression. Contributed by Weiwei Yang. 2018-02-26 12:15:16 -08:00
Jason Lowe
9497215084 YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:06:28 -06:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Wangda Tan
ca1043ab90 YARN-7328. ResourceUtils allows yarn.nodemanager.resource-types.memory-mb and .vcores to override yarn.nodemanager.resource.memory-mb and .cpu-vcores. (lovekesh bansal via wangda) 
Change-Id: Ibb1faf0beefec079dae2208986976b2f64650672
 2018-02-17 12:28:08 +08:00
Sunil G
aae629913c YARN-7292. Retrospect Resource Profile Behavior for overriding capability. Contributed by Wangda Tan. 2018-02-16 12:37:40 +05:30
Konstantinos Karanasos
0b489e564c YARN-7920. Simplify configuration for PlacementConstraints. Contributed by Wangda Tan. 2018-02-15 14:23:38 -08:00
Eric Payne
bddfe42e2c Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit c5e6e3de1c.
 2018-02-14 14:29:20 -06:00
Eric Payne
c5e6e3de1c YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-13 10:11:02 -06:00
Wangda Tan
e795833d8c YARN-7906. Fix mvn site fails with error: Multiple sources of package comments found for package o.a.h.y.client.api.impl. (Akira Ajisaka via wangda) 
Change-Id: I20221d97446e97f208d587eacbc60448c11ffd48
 2018-02-12 10:25:22 +08:00
Weiwei Yang
a08c048832 YARN-7838. Support AND/OR constraints in Distributed Shell. Contributed by Weiwei Yang. 2018-02-11 14:20:46 +08:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Arun Suresh
8df7666fe1 YARN-7780. Documentation for Placement Constraints. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
8779a35742 YARN-7709. Remove SELF from TargetExpression type. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
69de9a1ba9 YARN-7448. [API] Add SchedulingRequest to the AllocateRequest. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos
db928556c8 YARN-6595. [API] Add Placement Constraints at the application level. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Wangda Tan
b57e8bc300 YARN-6594. [API] Introduce SchedulingRequest object. (Konstantinos Karanasos via wangda) 2018-01-31 01:30:17 -08:00
Wangda Tan
33a796d9b7 YARN-6593. [API] Introduce Placement Constraint object. (Konstantinos Karanasos via wangda) 
Change-Id: Id00edb7185fdf01cce6e40f920cac3585f8cbe9c
 2018-01-31 01:30:17 -08:00
Haibo Chen
649ef7ac33 YARN-7064. Use cgroup to get container resource utilization. (Miklos Szegedi via Haibo Chen) 2018-01-26 16:27:31 -08:00
Eric Yang
97fe3cc187 YARN-7729. Add support for setting Docker PID namespace mode. (Contributed by Billie Rinaldi) 2018-01-22 16:33:38 -05:00
Sunil G
06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Rohith Sharma K S
d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Wangda Tan
edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Eric Yang
3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Wangda Tan
a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter
f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Varun Saxena
c0aeb666a4 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 2017-12-19 22:29:24 +05:30
Akira Ajisaka
001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Sunil G
2564b4d07f YARN-7119. Support multiple resource types in rmadmin updateNodeResource command. Contributed by Manikandan R. 2017-12-14 16:11:57 +05:30
Sunil G
10fc8d2a7d YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 2017-12-13 22:00:07 +05:30
Wangda Tan
d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
0311cf0535 YARN-7381. Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml. (Xuan Gong via wangda) 
Change-Id: I1ed58dafad5cc276eea5c0b0813cf04f57d73a87
 2017-12-05 13:09:49 -08:00
Jason Lowe
25df505421 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 2017-12-01 15:37:36 -06:00
Wangda Tan
7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Daniel Templeton
8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Yufei Gu
d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Jian He
fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Arun Suresh
b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Sunil G
aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Haibo Chen
67bbbe1c0c YARN-7531. ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType(). 2017-11-21 09:09:16 -08:00
Subru Krishnan
d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Varun Vasudev
5f0b238a11 YARN-7430. Enable user re-mapping for Docker containers by default. Contributed by Eric Yang. 2017-11-17 12:04:47 +05:30
Daniel Templeton
e094eb74b9 YARN-6953. Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() 
(Contributed by Manikandan R via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Jian He
5323b0048b YARN-7466. ResourceRequest has a different default for allocationRequestId than Container. Contributed by Chandni Singh 2017-11-13 15:37:39 -08:00
Daniel Templeton
2e512f016e YARN-7442. [YARN-7069] Limit format of resource type name (Contributed by Wangda Tan via Daniel Templeton) 2017-11-13 11:03:30 -08:00
bibinchundatt
2c2b7a3672 YARN-7406. Moving logging APIs over to slf4j in hadoop-yarn-api. Contributed by Yeliang Cang. 2017-11-11 10:36:27 +05:30
Wangda Tan
dd07038ffa YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-09 14:51:15 -08:00
Daniel Templeton
462f6c490e YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
 2017-11-09 11:58:49 -08:00
Jian He
c723021579 YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He
673c0db43c Revert "YARN-6626. Embed REST API service into RM. Contributed by Eric Yang" 
This reverts commit 63d1084e9781e0fee876916190b69f6242dd00e4.
 2017-11-06 13:30:17 -08:00
Jian He
9e677fa05c YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Sunil G
1700adc6f7 YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. Contributed by Wangda Tan. 2017-11-03 07:25:29 +05:30
Daniel Templeton
d64736d589 YARN-7336. Unsafe cast from long to int Resource.hashCode() method 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-10-30 12:41:28 -07:00
Daniel Templeton
9a7e810838 YARN-6927. Add support for individual resource types requests in MapReduce 
(Contributed by Gergo Repas via Daniel Templeton)
 2017-10-30 11:04:22 -07:00
Daniel Templeton
9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00
Sunil G
9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Jason Lowe
665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Wangda Tan
36e158ae98 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 
Change-Id: I14c5ea7252b7c17e86ab38f692b5f9d43196dbe0
 2017-10-26 20:15:19 -07:00
Robert Kanter
b1de78619f YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 2017-10-26 17:47:32 -07:00
Subru Krishnan
25932da6d1 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 2017-10-26 12:10:14 -07:00
Yufei Gu
7a27c2c367 YARN-7270. Fix unsafe casting from long to int for class Resource and 
its sub-classes. (Yufei)
 2017-10-13 12:41:59 -07:00
Wangda Tan
8bcc49e677 YARN-7205. Log improvements for the ResourceUtils. (Sunil G via wangda) 
Change-Id: I0f5b7a7f68ec5d3e1d52211f83fdd089bc0bfd37
 2017-10-11 15:25:28 -07:00
Sunil G
fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Wangda Tan
5cfbad68c0 YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) 
Change-Id: I959c8531bd48231956068cb7d2312f3235549f9c
(cherry picked from commit 8536db5a7549fa44d9f9e444b41bd5978b5c6ae1)
 2017-10-09 20:36:57 -07:00
Jonathan Hung
09c5dfe937 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:11:23 -07:00
Wangda Tan
ff39c0de20 YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:11:23 -07:00
Xuan
74ba6ffa0b YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Wangda Tan
a4e6253046 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:11:23 -07:00
Jonathan Hung
ef686635c1 YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:11:23 -07:00
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Arun Suresh
b733348dde YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 2017-10-05 09:58:04 -07:00
Ray Chiang
67e12169b0 YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang) 2017-10-03 15:06:07 -07:00
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Subru Krishnan
d4d2fd1acd YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru). 2017-10-02 18:14:44 -07:00
Jason Lowe
85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Jason Lowe
bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Arun Suresh
56ef5279c1 YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 2017-09-18 15:49:31 -07:00
Jason Lowe
a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Yufei Gu
09b476e6da YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 2017-09-14 11:23:37 -07:00
Arun Suresh
4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
864fbacd45 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh) 2017-09-14 08:48:39 -07:00
Rohith Sharma K S
5324388cf2 YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-13 23:36:47 +05:30
Sunil G
da0b6a354b YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Daniel Templeton
bf2b687412 YARN-7136. Additional Performance Improvement for Resource Profile Feature 
(Contributed by Wangda Tan via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Sunil G
805095496d YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R. 2017-09-12 09:19:12 -07:00
Wangda Tan
53df3eac50 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda) 
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
 2017-09-12 09:19:12 -07:00
Wangda Tan
e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Sunil G
a333ba54e3 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Wangda Tan
39240b61a1 YARN-7043. Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
 2017-09-12 09:19:12 -07:00
Sunil G
758b771943 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 2017-09-12 09:19:12 -07:00
Wangda Tan
df3855541a YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
 2017-09-12 09:19:12 -07:00
Sunil G
a6989af1d9 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Sunil G
2b51b262ab YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 2017-09-12 09:19:12 -07:00
Yufei Gu
d5e9939ebb YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 2017-09-12 09:19:11 -07:00
Daniel Templeton
3aeaafecb8 YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Sunil G
ca2808d8bb YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
58da54640e YARN-6445. [YARN-3926] Performance improvements in resource profile branch with respect to SLS. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
dae65f3bef YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
7805deed48 YARN-5588. [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Arun Suresh
6708ac3301 YARN-5587. Add support for resource profiles. (vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev
c2032e251e YARN-5708. Implement APIs to get resource profiles from the RM. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev
a9b4426302 YARN-5707. Add manager class for resource profiles. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S
9e4ba6aff5 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev
759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Arun Suresh
7ba698997b YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev
ceb12c59a3 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Wangda Tan
1bbab7c157 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 2017-09-12 09:19:09 -07:00
Rohith Sharma K S
4a83170be4 YARN-7140. CollectorInfo should have Public visibility. Contributed by Varun Saxena. 2017-09-08 20:23:16 +05:30
Sunil G
56d93d2e39 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 07:15:17 +05:30
Jason Lowe
b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Subru Krishnan
b6e7d13696 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-06 16:46:01 -07:00
Subru Krishnan
e3345e985b Revert "Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem." 
This reverts commit 7996eca7dc.
 2017-09-06 16:39:23 -07:00
Subru Krishnan
7996eca7dc Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-01 15:16:40 -07:00
Oleg Danilov
c5281a85e1
YARN-7023. Incorrect ReservationId.compareTo() implementation. 
This closes #262

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2017-09-01 21:33:31 +02:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Jian He
08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Haibo Chen
9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Rohith Sharma K S
32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jason Lowe
d5ff965fee YARN-6820. Restrict read access to timelineservice v2 data. Contributed by Vrushali C 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Sangjin Lee
580d884913 YARN-6094. Update the coprocessor to be a dynamically loaded one. Contributed by Vrushali C. 2017-08-30 11:29:52 +05:30
Varun Saxena
02a9710a09 YARN-5585. [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee
ecbc8d4816 YARN-5922. Remove direct references of HBaseTimelineWriter/Reader in core ATS classes. Contributed by Haibo Chen. 
(cherry picked from commit a5a55a54ab)
 2017-08-30 11:29:51 +05:30
Sangjin Lee
643a20a358 YARN-5265. Make HBase configuration for the timeline service configurable. Contributed by Joep Rottinghuis. 2017-08-30 11:29:51 +05:30
Sangjin Lee
05ff04439e YARN-5715. Introduce entity prefix for return and sort order. Contributed by Rohith Sharma K S. 2017-08-30 11:29:51 +05:30
Varun Saxena
9bb5cc3d49 YARN-3649. Allow configurable prefix for hbase table names like prod, exp, test etc (Vrushali C via Varun Saxena) 2017-08-30 11:29:50 +05:30
Li Lu
e72e6260da YARN-5229. Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl. (Vrushali C via gtcarrera9) 2017-08-30 11:29:50 +05:30
Carlo Curino
cc8893edc0 YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-29 14:53:09 -07:00
Junping Du
c2cb7ea1ef YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 2017-08-24 13:36:49 -07:00
Arun Suresh
8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Subru Krishnan
de462da04e YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 2017-08-16 11:43:24 -07:00
Carlo Curino
cc59b5fb26 YARN-6896. Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-11 15:58:01 -07:00
Subru Krishnan
bbbf0e2a41 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-11 13:58:45 -07:00
Junping Du
f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh
35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Carlo Curino
acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec860819b080ccc82454480a436c0adc)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
 2017-08-01 17:28:24 -07:00
Carlo Curino
5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
 2017-08-01 17:28:23 -07:00
Subru Krishnan
4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
 2017-08-01 17:28:21 -07:00