big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,060 Commits 2 Branches 2 Tags 264 MiB
1c9d2ab503
Commit Graph

1369 Commits

Author SHA1 Message Date
Arun Suresh
99e5dd68d0 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh) 2016-06-22 19:04:54 -07:00
Tsuyoshi Ozawa
5d58858bb6 HADOOP-9613. [JDK8] Update jersey version to latest 1.x release. 2016-06-21 08:05:32 +09:00
Junping Du
d0162f2040 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev. 2016-06-19 17:44:54 -07:00
Karthik Kambatla
20f2799938 YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha) 2016-06-17 22:24:42 -07:00
Karthik Kambatla
fbbe0bb627 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit. Addendum to fix javac warning. (Arun Suresh via kasha) 2016-06-17 22:12:50 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Rohith Sharma K S
28b66ae919 YARN-4989. TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently. Contributed by Ajith S. 2016-06-13 11:09:32 +05:30
Arun Suresh
5279af7cd4 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh) 2016-06-10 22:33:42 -07:00
Rohith Sharma K S
e0f4620cc7 YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe. 2016-06-11 10:22:27 +05:30
Wangda Tan
244506f9c8 YARN-5208. Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled. (Rohith Sharma K S via wangda) 2016-06-10 09:34:32 -07:00
Wangda Tan
620325e816 YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-06-07 15:06:42 -07:00
Arun Suresh
3a154f75ed YARN-4525. Fix bug in RLESparseResourceAllocation.getRangeOverlapping(). (Ishai Menache and Carlo Curino via asuresh) 2016-06-06 21:18:32 -07:00
Arun Suresh
7a9b7372a1 YARN-5185. StageAllocaterGreedyRLE: Fix NPE in corner case. (Carlo Curino via asuresh) 2016-06-06 21:06:52 -07:00
Ming Ma
4a1cedc010 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma) 2016-06-06 14:30:51 -07:00
Arun Suresh
db54670e83 YARN-5165. Fix NoOvercommitPolicy to take advantage of RLE representation of plan. (Carlo Curino via asuresh) 2016-06-03 14:49:32 -07:00
Vinod Kumar Vavilapalli
f10ebc67f5 YARN-5098. Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time. Contributed by Jian He. 2016-06-03 13:00:07 -07:00
Jian He
097baaaeba YARN-1815. Work preserving recovery of Unmanged AMs. Contributed by Subru Krishnan 2016-06-03 10:49:30 -07:00
Arun Suresh
dc26601d8f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh) 2016-06-02 09:01:02 -07:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Arun Suresh
aa975bc781 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh) 2016-05-27 14:06:32 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Rohith Sharma K S
0a544f8a3e YARN-5005. TestRMWebServices#testDumpingSchedulerLogs fails randomly. Contributed by Bibin A Chundatt. 2016-05-27 10:44:35 +05:30
Arun Suresh
5b41b288d0 YARN-5162. Fix Exceptions thrown during in registerAM call when Distributed Scheduling is Enabled (Hitesh Sharma via asuresh) 2016-05-26 14:56:37 -07:00
Karthik Kambatla
04ded558b0 YARN-5035. FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on. (kasha) 2016-05-26 14:41:07 -07:00
Karthik Kambatla
4f513a4a8e YARN-4866. FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy. (Yufei Gu via kasha) 2016-05-25 22:13:27 -07:00
Carlo Curino
013532a95e YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino) 2016-05-25 16:55:49 -07:00
Rohith Sharma K S
28bd63e92b YARN-5024. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers random failure. Contributed by Bibin A Chundatt 2016-05-25 10:15:50 +05:30
Naganarasimha
edd716e99c YARN-5114. Add additional tests in TestRMWebServicesApps and rectify testInvalidAppAttempts failure in 2.8. Contributed by Bibin A Chundatt 2016-05-25 06:11:38 +08:00
Karthik Kambatla
f979d779e1 YARN-4878. Expose scheduling policy and max running apps over JMX for Yarn queues. (Yufei Gu via kasha) 2016-05-24 10:54:11 -07:00
Naganarasimha
b4078bd17b YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt 2016-05-24 08:06:53 +08:00
Karthik Kambatla
6d043aa4cf YARN-4979. FSAppAttempt demand calculation considers demands at multiple locality levels different. (Zhihai Xu via kasha) 2016-05-23 14:29:28 -07:00
Jason Lowe
ac954486c5 YARN-5055. max apps per user can be larger than max per queue. Contributed by Eric Badger 2016-05-23 15:54:42 +00:00
Junping Du
22fcd819f0 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 
(cherry picked from commit 2703ec68712279494d67b0d76b7ac10e7a1628be)
 2016-05-19 14:15:21 -07:00
Jian He
feb90ffcca YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong 2016-05-19 13:01:36 -07:00
Arun Suresh
1597630681 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh) 2016-05-18 18:46:00 -07:00
Arun Suresh
8a9ecb7584 YARN-5090. Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster. (asuresh) 2016-05-17 19:01:29 -07:00
Jian He
fa3bc3405d YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 12:52:19 -07:00
Arun Suresh
ccc93e7812 YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh) 2016-05-16 23:22:01 -07:00
Eric Payne
1217c8f6b4 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger. 2016-05-16 20:28:04 +00:00
Arun Suresh
f45bc5a83e YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh) 2016-05-15 17:54:34 -07:00
Arun Suresh
f0ac18d001 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh) 2016-05-13 13:38:36 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Jason Lowe
013000fbc2 YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger 2016-05-12 20:28:36 +00:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Rohith Sharma K S
b7ac85259c YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-12 15:17:49 +05:30
Karthik Kambatla
4b4e4c6ba8 YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha) 2016-05-11 17:36:21 -07:00
Junping Du
39f2bac38b YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong. 2016-05-11 09:28:35 -07:00
Naganarasimha
2750fb900f YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt 2016-05-08 22:49:25 +05:30
Yongjun Zhang
47c41e7ac7 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang) 2016-05-06 21:50:09 -07:00
Jason Lowe
b2ed6ae731 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena 2016-05-06 22:59:39 +00:00
Wangda Tan
23248f63aa getApplicationReport call may raise NPE for removed queues. (Jian He via wangda) 2016-05-06 15:30:45 -07:00
Jian He
bb62e05925 YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan 2016-05-05 12:56:21 -07:00
Jason Lowe
d0da13229c YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-05-05 14:07:54 +00:00
Rohith Sharma K S
75e0450593 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt 2016-05-04 09:58:26 +05:30
Jason Lowe
ed54f5f1ff YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts 2016-05-03 20:03:41 +00:00
Jian He
dd80042c42 YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:27:25 -07:00
Karthik Kambatla
185c3d4de1 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha) 2016-04-27 09:43:23 -07:00
Jian He
4beff01354 YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode 2016-04-26 21:00:17 -07:00
Arun Suresh
341888a0aa YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh) 2016-04-26 20:12:12 -07:00
Karthik Kambatla
4b1dcbbe0c YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled 2016-04-26 05:10:09 -07:00
Arun Suresh
c282a08f38 YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh) 
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
 2016-04-24 22:38:33 -07:00
Wangda Tan
7cb3a3da96 YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda) 2016-04-22 11:40:32 -07:00
Eric Payne
3dce486d88 YARN-4556. TestFifoScheduler.testResourceOverCommit fails. Contributed by Akihiro Suda 2016-04-21 21:16:47 +00:00
Li Lu
7c6339f66a YARN-4968. A couple of AM retry unit tests need to wait SchedulerApplicationAttempt stopped. (Wangda Tan via gtcarrera9) 2016-04-21 13:25:33 -07:00
Karthik Kambatla
170c4fd4cd YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha) 2016-04-20 23:58:12 -07:00
Wangda Tan
33fd95a99c YARN-4890. Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate. (Sunil G via wangda) 2016-04-20 17:37:38 -07:00
Wangda Tan
fdc46bfb37 YARN-4934. Reserved Resource for QueueMetrics needs to be handled correctly in few cases. (Sunil G via wangda) 2016-04-16 22:47:41 -07:00
Jason Lowe
69f3d428d5 YARN-4940. yarn node -list -all failed if RM start with decommissioned node. Contributed by sandflee 2016-04-15 20:36:45 +00:00
Jason Lowe
2a5da97f81 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 1cbcd4a491.
 2016-04-11 15:51:01 +00:00
Akira Ajisaka
1ff27f9d12 YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-04-11 14:55:03 +09:00
Karthik Kambatla
ff95fd547b YARN-4927. TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler. (Bibin A Chundatt via kasha) 2016-04-09 10:31:02 -07:00
Wangda Tan
ec06957941 YARN-3215. Respect labels in CapacityScheduler when computing headroom. (Naganarasimha G R via wangda) 2016-04-08 15:33:04 -07:00
Jian He
9cb0c963d2 YARN-4740. AM may not receive the container complete msg when it restarts. Contributed by Jun Gong 2016-04-08 11:20:35 -07:00
Jian He
93bacda08b YARN-4769. Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI. Contributed by Varun Vasudev 2016-04-06 16:13:47 -07:00
Wangda Tan
21eb428448 YARN-4699. Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node. (Sunil G via wangda) 2016-04-05 16:24:11 -07:00
Junping Du
6be28bcc46 YARN-4893. Fix some intermittent test failures in TestRMAdminService. Contributed by Brahma Reddy Battula. 2016-04-05 06:57:54 -07:00
Jason Lowe
1cbcd4a491 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-04-05 13:40:19 +00:00
Rohith Sharma K S
776b549e2a YARN-4609. RM Nodes list page takes too much time to load. Contributed by Bibin A Chundatt 2016-04-05 14:47:25 +05:30
Rohith Sharma K S
552237d4a3 YARN-4880. Running TestZKRMStateStorePerf with real zookeeper cluster throws NPE. Contributed by Sunil G 2016-04-05 14:26:19 +05:30
naganarasimha
5092c94195 YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt 2016-04-04 16:25:03 +05:30
Rohith Sharma K S
1e6f92977d YARN-4607. Pagination support for AppAttempt page TotalOutstandingResource Requests table. Contributed by Bibin A Chundatt 2016-04-04 08:09:29 +05:30
Wangda Tan
12b11e2e68 YARN-4634. Scheduler UI/Metrics need to consider cases like non-queue label mappings. (Sunil G via wangda) 2016-03-31 14:35:18 -07:00
Robert Kanter
7a021471c3 YARN-4639. Remove dead code in TestDelegationTokenRenewer added in YARN-3055 (templedf via rkanter) 2016-03-31 13:09:09 -07:00
Jian He
60e4116bf1 YARN-4822. Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates. Contributed by Wangda Tan 2016-03-30 12:43:52 -07:00
Wangda Tan
fc055a3cbe YARN-4865. Track Reserved resources in ResourceUsage and QueueCapacities. (Sunil G via wangda) 2016-03-29 17:07:55 -07:00
Jian He
524bc3c33a YARN-998. Keep NM resource updated through dynamic resource config for RM/NM restart. Contributed by Junping Du 2016-03-28 11:12:33 -07:00
Karthik Kambatla
49ff54c860 YARN-4805. Don't go through all schedulers in ParameterizedTestBase. (kasha) 2016-03-26 21:45:13 -07:00
Arun Suresh
00bebb7e58 YARN-4823. Refactor the nested reservation id field in listReservation to simple string field. (subru via asuresh) 2016-03-25 15:54:38 -07:00
Arun Suresh
d82e797b65 YARN-4825. Remove redundant code in ClientRMService::listReservations. (subru via asuresh) 2016-03-24 09:59:55 -07:00
Allen Wittenauer
b1394d6307 YARN-4850. test-fair-scheduler.xml isn't valid xml (Yufei Gu via aw) 2016-03-24 08:15:58 -07:00
Junping Du
19b645c938 YARN-4820. ResourceManager web redirects in HA mode drops query parameters. Contributed by Varun Vasudev. 2016-03-23 19:34:30 -07:00
Junping Du
ca8106d2dd YARN-4785. inconsistent value type of the type field for LeafQueueInfo in response of RM REST API. 2016-03-17 09:04:41 -07:00
Karthik Kambatla
f84af8bd58 YARN-4812. TestFairScheduler#testContinuousScheduling fails intermittently. (kasha) 2016-03-17 05:54:06 -07:00
Wangda Tan
ae14e5d07f YARN-4108. CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request. (Wangda Tan) 
(cherry picked from commit 7e8c9beb41)
 2016-03-16 17:02:33 -07:00
Wangda Tan
fa7a43529d Revert "CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request. (Wangda Tan)" 
This reverts commit 7e8c9beb41.
 2016-03-16 17:02:10 -07:00
Wangda Tan
7e8c9beb41 CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request. (Wangda Tan) 2016-03-16 16:59:59 -07:00
Karthik Kambatla
3ef5500783 YARN-4560. Make scheduler error checking message more user friendly. (Ray Chiang via kasha) 2016-03-15 23:45:01 -07:00
Karthik Kambatla
20d389ce61 YARN-4719. Add a helper library to maintain node state and allows common queries. (kasha) 2016-03-14 14:19:05 -07:00
Wangda Tan
0233d4e0ee YARN-4465. SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled. (Bibin A Chundatt via wangda) 2016-03-08 14:27:03 -08:00