big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9,001 Commits 2 Branches 2 Tags 264 MiB
30d56fdbb4
Commit Graph

679 Commits

Author SHA1 Message Date
Jian He
30d56fdbb4 YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. Contributed by Chen He and Craig Welch 2014-10-07 13:45:04 -07:00
Jian He
519e5a7dd2 YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when AM allocates. Contributed by Craig Welch 2014-10-06 15:48:46 -07:00
Jian He
ea26cc0b4a YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier to use protobuf as payload. Contributed by Junping Du 2014-10-06 10:47:43 -07:00
Chris Douglas
c8212bacb1 YARN-1051. Add a system for creating reservations of cluster capacity. 
Contributed by Subru Krishnan and Carlo Curino.
 2014-10-03 17:09:02 -07:00
subru
a2986234be YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched for branch YARN-1051. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit c47464aba407d1dafe10be23fe454f0489cc4367)
 2014-10-03 15:43:23 -07:00
subru
5e10a13bb4 YARN-2576. Making test patch pass in branch. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 90ac0be86b898aefec5471db4027554c8e1b310c)
 2014-10-03 15:43:13 -07:00
subru
6261f7cc69 YARN-2080. Integrating reservation system with ResourceManager and client-RM protocol. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 8baeaead8532898163f1006276b731a237b1a559)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
 2014-10-03 15:42:43 -07:00
subru
3418c56bcf YARN-1712. Plan follower that synchronizes the current state of reservation subsystem with the scheduler. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 169085319b8b76641f8b9f6840a3fef06d221e2b)
 2014-10-03 15:42:10 -07:00
carlo curino
b6df0dddcd YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit c4918cb4cb5a267a8cfd6eace28fcfe7ad6174e8)
 2014-10-03 15:42:03 -07:00
carlo curino
f66ffcf832 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899b37262773f3dc117157bb746bf8918)
 2014-10-03 15:41:57 -07:00
subru
cf4b34282a YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b958b1e602fd4ea4078ef8d4742a7c3)
 2014-10-03 15:41:51 -07:00
carlo curino
1c6950354f YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266f2c5e6eead554d8a331ed7e75e10d5)
 2014-10-03 15:41:21 -07:00
carlo curino
eb3e40b833 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7fecda9fc18ade34d633eca895865a70)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
 2014-10-03 15:41:02 -07:00
Karthik Kambatla
80d11eb68e YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 2014-10-03 11:49:49 -07:00
Jian He
054f285526 YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 2014-10-02 15:13:33 -07:00
Karthik Kambatla
5e0b49da9c YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 2014-10-02 10:15:04 -07:00
Zhijie Shen
52bbe0f11b YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 2014-10-01 15:38:11 -07:00
Jian He
bbff96be48 YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. Contributed by Zhijie Shen 2014-09-30 16:44:17 -07:00
Karthik Kambatla
14d60dadc2 YARN-2594. Potential deadlock in RM when querying ApplicationResourceUsageReport. (Wangda Tan via kasha) 2014-09-30 16:12:27 -07:00
Zhijie Shen
9582a50176 YARN-2627. Added the info logs of attemptFailuresValidityInterval and number of previous failed attempts. Contributed by Xuan Gong. 2014-09-30 12:57:45 -07:00
Jason Lowe
9c22065109 YARN-1769. CapacityScheduler: Improve reservations. Contributed by Thomas Graves 2014-09-29 14:12:18 +00:00
Jian He
5391919b09 YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier to use protobuf object as the payload. Contributed by Junping Du. 2014-09-26 17:48:41 -07:00
Karthik Kambatla
f4357240a6 YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and clock access. (Wei Yan via kasha) 2014-09-25 17:42:51 -07:00
Jason Lowe
8269bfa613 YARN-2523. ResourceManager UI showing negative value for "Decommissioned Nodes" field. Contributed by Rohith 2014-09-25 22:37:05 +00:00
Zhijie Shen
72b0881ca6 YARN-2546. Made REST API for application creation/submission use numeric and boolean types instead of the string of them. Contributed by Varun Vasudev. 2014-09-24 17:57:32 -07:00
Zhijie Shen
c86674a3a4 YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. Contributed by Xuan Gong. 2014-09-24 17:50:26 -07:00
Karthik Kambatla
39c87344e1 YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) 2014-09-24 15:23:31 -07:00
Karthik Kambatla
f5578207d2 YARN-2252. Intermittent failure of TestFairScheduler.testContinuousScheduling. (Ratandeep Ratti and kasha via kasha) 2014-09-23 00:03:16 -07:00
Karthik Kambatla
568d3dc2bb YARN-1959. Fix headroom calculation in FairScheduler. (Anubhav Dhoot via kasha) 2014-09-22 23:49:39 -07:00
Karthik Kambatla
43efdd30b5 YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. (Wei Yan via kasha) 2014-09-22 16:09:52 -07:00
Karthik Kambatla
23e17ce0ff YARN-2540. FairScheduler: Queue filters not working on scheduler page in RM UI. (Ashwin Shankar via kasha) 2014-09-22 15:40:47 -07:00
Jian He
0a641496c7 YARN-1372. Ensure all completed containers are reported to the AMs across RM restart. Contributed by Anubhav Dhoot 2014-09-22 10:30:53 -07:00
Karthik Kambatla
9721e2c1fe YARN-2453. TestProportionalCapacityPreemptionPolicy fails with FairScheduler. (Zhihai Xu via kasha) 2014-09-21 23:13:45 -07:00
Karthik Kambatla
c50fc92502 YARN-2452. TestRMApplicationHistoryWriter fails with FairScheduler. (Zhihai Xu via kasha) 2014-09-21 13:15:04 -07:00
Jian He
444acf8ea7 YARN-2565. Fixed RM to not use FileSystemApplicationHistoryStore unless explicitly set. Contributed by Zhijie Shen 2014-09-19 11:26:29 -07:00
Zhijie Shen
6fe5c6b746 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 2014-09-18 21:56:56 -07:00
Jason Lowe
a337f0e354 YARN-2561. MR job client cannot reconnect to AM after NM restart. Contributed by Junping Du 2014-09-18 21:34:40 +00:00
Jason Lowe
9ea7b6c063 YARN-2363. Submitted applications occasionally lack a tracking URL. Contributed by Jason Lowe 2014-09-18 20:13:16 +00:00
Vinod Kumar Vavilapalli
485c96e3cb YARN-2001. Added a time threshold for RM to wait before starting container allocations after restart/failover. Contributed by Jian He. 2014-09-18 11:03:12 -07:00
Jian He
ee21b13cbd YARN-2559. Fixed NPE in SystemMetricsPublisher when retrieving FinalApplicationStatus. Contributed by Zhijie Shen 2014-09-17 21:44:15 -07:00
junping_du
90a0c03f0a YARN-1250. Generic history service should support application-acls. (Contributed by Zhijie Shen) 2014-09-16 18:20:49 -07:00
Vinod Kumar Vavilapalli
14e2639fd0 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 2014-09-13 18:04:05 -07:00
XuanGong
e65ae575a0 YARN-2456. Possible livelock in CapacityScheduler when RM is recovering 
apps. Contributed by Jian He
 2014-09-12 15:21:46 -07:00
Jian He
3122daa802 YARN-2229. Changed the integer field of ContainerId to be long type. Contributed by Tsuyoshi OZAWA 2014-09-12 10:33:33 -07:00
Jason Lowe
78b048393a YARN-2484. FileSystemRMStateStore#readFile/writeFile should close FSData(In|Out)putStream in final block. Contributed by Tsuyoshi OZAWA 2014-09-12 14:09:14 +00:00
Zhijie Shen
469ea3dcef YARN-2538. Added logs when RM sends roll-overed AMRMToken to AM. Contributed by Xuan Gong. 2014-09-11 22:19:00 -07:00
junping_du
6b8b1608e6 YARN-2033. Merging generic-history into the Timeline Store (Contributed by Zhijie Shen) 2014-09-12 10:04:51 +08:00
Karthik Kambatla
c11ada5ea6 YARN-2534. FairScheduler: Potential integer overflow calculating totalMaxShare. (Zhihai Xu via kasha) 2014-09-11 12:06:06 -07:00
Jian He
83be3ad444 YARN-415. Capture aggregate memory allocation at the app-level for chargeback. Contributed by Eric Payne & Andrey Klochkov 2014-09-10 18:20:54 -07:00
Jian He
cbfe26370b YARN-2158. Fixed TestRMWebServicesAppsModification#testSingleAppKill test failure. Contributed by Varun Vasudev 2014-09-10 12:47:34 -07:00