big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,445 Commits 2 Branches 2 Tags 264 MiB
17e8cf501b
Commit Graph

2314 Commits

Author SHA1 Message Date
Giovanni Matteo Fumarola
da7f8c244d YARN-9431. Fix flaky junit test fair.TestAppRunnability after YARN-8967. Contributed by Wilfred Spiegelenburg. 2019-04-01 11:21:31 -07:00
Giovanni Matteo Fumarola
332cab5518 YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. 2019-04-01 11:06:51 -07:00
yufei
5257f50abb YARN-8967. Change FairScheduler to use PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-25 22:47:24 -07:00
Giovanni Matteo Fumarola
509b20b292 YARN-9404. TestApplicationLifetimeMonitor#testApplicationLifetimeMonitor fails intermittent. Contributed by Prabhu Joseph. 2019-03-22 11:45:39 -07:00
Zoltan Siegl
ce5eb9cb2e YARN-9358. Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 
(Contributed by Zoltan Siegl via Daniel Templeton)

Change-Id: I92d52c0ca630e71afb26b2b7587cbdbe79254a05
 2019-03-22 12:28:34 +01:00
Giovanni Matteo Fumarola
548997d6c9 YARN-9402. Opportunistic containers should not be scheduled on Decommissioning nodes. Contributed by Abhishek Modi. 2019-03-21 12:04:05 -07:00
Giovanni Matteo Fumarola
5d8bd0e5cb YARN-9392. Handle missing scheduler events in Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-03-19 11:00:21 -07:00
Eric Yang
09eabda314 YARN-9364. Remove commons-logging dependency from YARN. 
Contributed by Prabhu Joseph
 2019-03-18 19:58:42 -04:00
Eric Yang
5f6e225166 YARN-9363. Replaced debug logging with SLF4J parameterized log message. 
Contributed by Prabhu Joseph
 2019-03-18 13:57:18 -04:00
Eric Yang
2064ca015d YARN-9349. Changed logging to use slf4j api. 
Contributed by Prabhu Joseph
 2019-03-15 19:20:59 -04:00
Eric Yang
03f3c8aed2 YARN-4404. Corrected typo in javadoc. 
Contributed by Yesha Vora
 2019-03-15 18:04:04 -04:00
Eric Yang
39b4a37e02 YARN-9341. Fixed enentrant lock usage in YARN project. 
Contributed by Prabhu Joseph
 2019-03-07 16:47:45 -05:00
Vrushali C
491313ab84 YARN-8218 Add application launch time to ATSV1. Contributed by Abhishek Modi 2019-03-06 21:47:29 -08:00
Yufei Gu
0aefe2846f YARN-9298. Implement FS placement rules using PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-04 23:49:07 -08:00
Prabhu Joseph
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 14:10:08 +09:00
bibinchundatt
15098df744 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph." 
This reverts commit a63c358b78.
 2019-03-04 16:57:31 +05:30
Szilard Nemeth
538bb4880d YARN-9323. FSLeafQueue#computeMaxAMResource does not override zero values for custom resources 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: Id844ccf09488f367c0c7de0a3b2d4aca1bba31cc
 2019-02-27 19:59:48 -08:00
Szilard Nemeth
7b928f19a4 YARN-9322. Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: I14c12f1265999d62102f2ec5506d90015efeefe8
 2019-02-27 19:43:50 -08:00
Weiwei Yang
1779fc57a1 YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph. 2019-02-28 09:56:29 +08:00
Weiwei Yang
8c30114b00 YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie. 2019-02-27 17:29:02 +08:00
Rohith Sharma K S
8eae260af5 YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph. 2019-02-27 10:28:16 +05:30
Giovanni Matteo Fumarola
95372657fc YARN-9287. Consecutive StringBuilder append should be reuse. Contributed by Ayush Saxena. 2019-02-25 11:45:37 -08:00
Weiwei Yang
3e1739d589 YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang. 2019-02-26 00:08:13 +08:00
Weiwei Yang
9cd5c5447f YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph. 2019-02-24 22:42:27 +08:00
Weiwei Yang
50094d7fef YARN-9300. Lazy preemption should trigger an update on queue preemption metrics for CapacityScheduler. Contributed by Tao Yang. 2019-02-24 22:17:29 +08:00
bibinchundatt
ed13cf8406 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-22 21:01:34 +05:30
Weiwei Yang
9c88695bcd YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie. 2019-02-22 20:58:33 +08:00
Weiwei Yang
eedcc8e26e YARN-9315. TestCapacitySchedulerMetrics fails intermittently. Contributed by Prabhu Joseph. 2019-02-21 17:21:21 +08:00
bibinchundatt
a63c358b78 YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph. 2019-02-21 13:29:01 +05:30
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Giovanni Matteo Fumarola
7536488bbd YARN-996. REST API support for node resource configuration. Contributed by Inigo Goiri. 2019-02-11 15:27:43 -08:00
Weiwei Yang
0a1637c750 YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options. Contributed by Prabhu Joseph. 2019-02-11 15:53:50 +08:00
Rohith Sharma K S
7fa62e150c YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi. 2019-02-07 09:58:53 +05:30
Sunil G
604b2489a9 YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 2019-02-04 12:48:28 +05:30
Weiwei Yang
28ad20a711 YARN-9262. TestRMAppAttemptTransitions is failing with an NPE. Contributed by lujie. 2019-02-02 23:02:45 +08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Eric E Payne
04105bbfdb YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 2019-01-29 17:22:24 +00:00
Jonathan Hung
6cace58e21 YARN-9222. Print launchTime in ApplicationSummary 2019-01-25 13:17:00 -08:00
Eric Yang
a33ef4fd31 YARN-8867. Added resource localization status to YARN service status call. 
Contributed by Chandni Singh
 2019-01-24 18:43:21 -05:00
Weiwei Yang
a4bd64e724 YARN-9116. Capacity Scheduler: implements queue level maximum-allocation inheritance. Contributed by Aihua Xu. 2019-01-24 17:53:43 +08:00
Weiwei Yang
721d5c2a5f YARN-8101. Add UT to verify node-attributes in RM nodes rest API. Contributed by Prabhu Joseph. 2019-01-23 17:59:36 +08:00
Weiwei Yang
e3e076dc78 YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 2019-01-23 14:14:49 +08:00
Weiwei Yang
d43df31751 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 2019-01-22 10:23:32 +08:00
Weiwei Yang
abde1e1f58 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang. 2019-01-21 16:54:14 +08:00
Wangda Tan
6d7eedfd28 YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda) 
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
 2019-01-17 14:20:10 -08:00
Akira Ajisaka
614af50625
YARN-9179. Fix NPE in AbstractYarnScheduler#updateNewContainerInfo. 2019-01-15 12:59:59 +09:00
Jason Lowe
6a923464af YARN-6523. Optimize system credentials sent in node heartbeat responses. Contributed by Manikandan R 2019-01-08 16:54:05 -06:00
Weiwei Yang
0921b706f7 YARN-9037. [CSI] Ignore volume resource in resource calculators based on tags. Contributed by Sunil Govindan. 2019-01-08 14:30:53 +08:00
Weiwei Yang
944cf87223 YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg. 2019-01-07 15:57:31 +08:00
Sunil G
f4906ac019 YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang. 2019-01-04 12:10:00 +05:30
Weiwei Yang
cfe89e6f96 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie. 2019-01-03 23:56:28 +08:00
Akira Ajisaka
128f340f4f
YARN-8894. Improve InMemoryPlan#toString. Contributed by BELUGA BEHR. 2018-12-28 17:43:47 +09:00
Akira Ajisaka
15fe1b34dd
YARN-8878. Remove StringBuffer from ManagedParentQueue.java. Contributed by BELUGA BEHR. 2018-12-28 17:29:22 +09:00
Erik Krogen
091ad974cd HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Eric Yang
f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Weiwei Yang
f659485ee8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 10:56:42 +08:00
Haibo Chen
881230da21 YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth) 2018-12-11 11:41:32 -08:00
Haibo Chen
64411a6ff7 YARN-8738. FairScheduler should not parse negative maxResources or minResources values as positive. (Contributed by Szilard Nemeth) 2018-12-10 11:06:50 -08:00
Jonathan Hung
cc51607ccd YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 2018-12-07 10:45:37 -08:00
Haibo Chen
5d4a432660 YARN-9025. TestFairScheduler#testChildMaxResources is flaky. (Contributed by Szilard Nemeth) 2018-12-05 16:36:13 -08:00
Haibo Chen
b3c75c1f1d YARN-8985. Improve debug log in FSParentQueue when assigning container. (Contributed by Wilfred Spiegelenburg) 2018-12-05 15:02:57 -08:00
Haibo Chen
2bba329c14 YARN-8994. Fix race condition between move app and queue cleanup in Fair Scheduler. (Contributed by Wilfred Spiegelenburg) 2018-12-05 14:38:15 -08:00
Yufei Gu
e89941fdbb YARN-9041. Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt. Contributed by Wanqiang Ji. 2018-12-04 10:08:45 -08:00
Jonathan Hung
6d7b44c489 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:13:52 -08:00
bibinchundatt
07142f54a8 YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T. 2018-11-29 22:02:59 +05:30
bibinchundatt
a68d766e87 YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt. 2018-11-29 21:43:34 +05:30
Haibo Chen
a41b648e98 YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg) 2018-11-20 14:12:37 -08:00
Sunil G
5fb14e0635 YARN-8953. [CSI] CSI driver adaptor module support in NodeManager. Contributed by Weiwei Yang. 2018-11-20 08:02:11 +05:30
Weiwei Yang
d027a24f03 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 2018-11-18 23:18:26 +08:00
Weiwei Yang
c741109522 YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt. 2018-11-12 18:18:23 +08:00
Sunil G
4e7284443e YARN-8902. [CSI] Add volume manager that manages CSI volume lifecycle. Contributed by Weiwei Yang. 2018-11-12 11:57:02 +05:30
Haibo Chen
524a7523c4 YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg) 2018-11-08 16:02:48 -08:00
Weiwei Yang
e1bbf7dcdf YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang. 2018-11-08 17:47:18 +08:00
Weiwei Yang
c96cbe8659 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji. 2018-11-07 22:42:22 +08:00
Akira Ajisaka
951c98f890
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-07 11:17:35 +09:00
Jason Lowe
ab6aa4c726 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 2018-11-06 08:40:59 -06:00
Weiwei Yang
5d6554c722 YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang. 2018-11-06 14:50:09 +08:00
Weiwei Yang
c7fcca0d7e YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji. 2018-11-06 13:14:57 +08:00
Giovanni Matteo Fumarola
babc946d40 YARN-8954. Reservations list field in ReservationListInfo is not accessible. Contributed by Oleksandr Shevchenko. 2018-11-02 11:10:08 -07:00
Jonathan Hung
2ab611d48b YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 2018-11-01 14:22:00 -07:00
Weiwei Yang
1d90a0dd23 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 2018-10-29 11:53:10 +08:00
Jason Lowe
93fb3b4b9c YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg 2018-10-23 12:49:15 -05:00
Haibo Chen
285d2c0753 YARN-8449. RM HA for AM web server HTTPS Support. (Contributed by Robert Kanter) 2018-10-18 21:24:36 -07:00
Weiwei Yang
13cc0f50ea YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 2018-10-19 09:55:59 +08:00
Wangda Tan
780be14f07 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
 2018-10-18 10:58:21 -07:00
Sunil G
5085e5fa9e YARN-8759. Copy of resource-types.xml is not deleted if test fails, causes other test failures. Contributed by Antal Bálint Steinbach. 2018-10-17 16:05:08 +05:30
Haibo Chen
84e22a6af4 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 2018-10-16 14:14:30 -07:00
Haibo Chen
c2288ac45b YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter) 2018-10-16 13:36:26 -07:00
Weiwei Yang
edce866489 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan. 2018-10-10 09:32:27 +08:00
Haibo Chen
7c13872cbb YARN-8659. RMWebServices returns only RUNNING apps when filtered with queue. (Contributed by Szilard Nemeth) 2018-10-08 09:40:37 -07:00
Jason Lowe
9bb2801e8c YARN-4254. ApplicationAttempt stuck for ever due to UnknownHostException. Contributed by Bibin A Chundatt 2018-10-05 15:52:46 -05:00
Haibo Chen
c968365650 YARN-8644. Improve unit test for RMAppImpl.FinalTransition. (Contributed by Szilard Nemeth) 2018-10-05 09:32:30 -07:00
Haibo Chen
e60b797c88 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 2018-10-04 13:00:31 -07:00
Haibo Chen
b6d5d84e07 YARN-8732. Add unit tests of min/max allocation for custom resource types in FairScheduler. (Contributed by Szilard Nemeth) 2018-10-04 12:47:31 -07:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Haibo Chen
d0ee6fbe28 YARN-8621. Add test coverage of custom Resource Types for the apps/<appId> REST API endpoint. (Contributed by Szilard Nemeth) 2018-10-01 14:46:42 -07:00
Weiwei Yang
fd6be5898a YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach. 2018-09-29 17:47:12 +08:00
Eric E Payne
8598b498bc YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 2018-09-28 15:32:07 +00:00
Jason Lowe
6b988d821e YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang 2018-09-26 14:43:00 -07:00
Rohith Sharma K S
e5287a4fe0 YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt. 2018-09-26 12:30:26 +05:30