Chris Douglas
d383bfdcd4
YARN-5164. Use plan RLE to improve CapacityOverTimePolicy efficiency
2016-07-25 16:37:50 -07:00
Akira Ajisaka
c63afdbe14
YARN-4883. Make consistent operation name in AdminService. Contributed by Kai Sasaki.
2016-07-20 16:51:01 -07:00
Arun Suresh
cda0a280dd
YARN-5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh)
2016-07-19 10:43:37 -07:00
Arun Suresh
5f2d33a551
Revert "YARN=5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh)"
...
This reverts commit e905a42a2c
.
2016-07-19 10:43:19 -07:00
Ray Chiang
f5f1c81e7d
YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang)
2016-07-15 14:38:50 -07:00
Arun Suresh
e905a42a2c
YARN=5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh)
2016-07-15 14:35:12 -07:00
Wangda Tan
24db9167f1
YARN-4484. Available Resource calculation for a queue is not correct when used with labels. (Sunil G via wangda)
2016-07-15 11:40:12 -07:00
Varun Saxena
06c56ff79b
YARN-5353. ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena).
2016-07-13 07:55:34 +05:30
Jian He
819224dcf9
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:36:20 -07:00
Sangjin Lee
6cf6ab7b78
Made a number of miscellaneous fixes for javac, javadoc, and checstyle warnings.
2016-07-10 08:46:05 -07:00
Vrushali
6d943038f6
Cleanup changes during rebase with trunk (Vrushali C)
2016-07-10 08:46:04 -07:00
Varun Saxena
1ff6833bba
YARN-5243. fix several rebase and other miscellaneous issues before merge. (Sangjin Lee via Varun Saxena)
2016-07-10 08:46:03 -07:00
Sangjin Lee
702236129b
YARN-5095. flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled (Varun Saxena via sjlee)
2016-07-10 08:46:00 -07:00
Sangjin Lee
a1b6d7456f
YARN-5018. Online aggregation logic should not run immediately after collectors got started (Li Lu via sjlee)
2016-07-10 08:45:59 -07:00
Li Lu
c2055a97d5
YARN-3150. Documenting the timeline service v2. (Sangjin Lee and Vrushali C via gtcarrera9)
2016-07-10 08:45:57 -07:00
Varun Saxena
a3cf40e532
YARN-3461. Consolidate flow name/version/run defaults. (Sangjin Lee via Varun Saxena)
2016-07-10 08:45:55 -07:00
Sangjin Lee
960af7d471
YARN-4409. Fix javadoc and checkstyle issues in timelineservice code (Varun Saxena via sjlee)
2016-07-10 08:45:53 -07:00
Naganarasimha
06f0b50a28
YARN-4644. TestRMRestart fails and findbugs issue in YARN-2928 branch (Varun Saxena via Naganarasimha G R)
2016-07-10 08:45:52 -07:00
Li Lu
34f02f07d5
Rebase to latest trunk
2016-07-10 08:45:51 -07:00
Varun Saxena
829cceebc0
YARN-3586. RM to only get back addresses of Collectors that NM needs to know.
...
(Junping Du via Varun Saxena).
2016-07-10 08:45:50 -07:00
Varun Saxena
c4d7bbda5c
YARN-4460. [Bug fix] RM fails to start when SMP is enabled. (Li Lu via Varun Saxena)
2016-07-10 08:45:49 -07:00
Xuan
2e2dbf59d1
YARN-4392. ApplicationCreatedEvent event time resets after RM
...
restart/failover. Contributed by Naganarasimha G R and Xuan Gong
(cherry picked from commit 4546c7582b
)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
2016-07-10 08:45:49 -07:00
Li Lu
89e5c44f9e
YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has no
...
impact when it's turned off. Contributed by Sangjin Lee.
2016-07-10 08:45:48 -07:00
Sangjin Lee
10ec5586fb
YARN-4129. Refactor the SystemMetricPublisher in RM to better support newer events (Naganarasimha G R via sjlee)
2016-07-10 08:45:46 -07:00
Sangjin Lee
8d9476ec5f
YARN-4058. Miscellaneous issues in NodeManager project (Naganarasimha G R via sjlee)
2016-07-10 08:45:43 -07:00
Sangjin Lee
22e7ae5771
YARN-3792. Test case failures in TestDistributedShell and some issue fixes related to ATSV2 (Naganarasimha G R via sjlee)
...
(cherry picked from commit 84f37f1c7eefec6d139cbf091c50d6c06f734323)
2016-07-10 08:45:38 -07:00
Zhijie Shen
f3c661e8dd
YARN-3044. Made RM write app, attempt and optional container lifecycle events to timeline service v2. Contributed by Naganarasimha G R.
2016-07-10 08:45:37 -07:00
Sangjin Lee
dc1f306fdc
YARN-3562. unit tests failures and issues found from findbug from earlier ATS checkins (Naganarasimha G R via sjlee)
2016-07-10 08:45:35 -07:00
Sangjin Lee
11e8905d8d
YARN-3390. Reuse TimelineCollectorManager for RM (Zhijie Shen via sjlee)
...
(cherry picked from commit 58221188811e0f61d842dac89e1f4ad4fd8aa182)
2016-07-10 08:45:33 -07:00
Junping Du
47f35a30bb
YARN-3391. Clearly define flow ID/ flow run / flow version in API and storage. Contributed by Zhijie Shen
...
(cherry picked from commit 68c6232f8423e55b4d152ef3d1d66aeb2d6a555e)
2016-07-10 08:45:33 -07:00
Zhijie Shen
5712b8f9fd
YARN-3334. NM uses timeline client to publish container metrics to new timeline service. Contributed by Junping Du.
2016-07-10 08:45:33 -07:00
Junping Du
d67c9bdb4d
YARN-3040. Make putEntities operation be aware of the app's context. Contributed by Zhijie Shen
2016-07-10 08:45:32 -07:00
Junping Du
5e3d9a477b
YARN-3034. Implement RM starting its timeline collector. Contributed by Naganarasimha G R
2016-07-10 08:45:32 -07:00
Junping Du
2188a07e5b
YARN-3333. Rename TimelineAggregator etc. to TimelineCollector. Contributed by Sangjin Lee
2016-07-10 08:45:31 -07:00
Zhijie Shen
9b56364080
YARN-3039. Implemented the app-level timeline aggregator discovery service. Contributed by Junping Du.
2016-07-10 08:45:31 -07:00
Varun Saxena
5252562edf
YARN-5297. Avoid printing a stack trace when recovering an app after the RM restarts. (Junping Du via Varun Saxena).
2016-07-09 00:09:25 +05:30
Junping Du
30ee57ceb1
YARN-4939. The decommissioning Node should keep alive during NM restart. Contributed by sandflee.
2016-07-08 04:14:53 -07:00
Wangda Tan
04f6ebb66a
YARN-5294. Pass remote ip address down to YarnAuthorizationProvider. (Jian He via wangda)
2016-07-06 10:36:48 -07:00
Varun Saxena
8e672e3c71
YARN-5286. Add RPC port info in RM web service's response when getting app status. (Jun Gong via Varun Saxena).
2016-07-05 22:56:07 +05:30
Rohith Sharma K S
26b5e6116f
YARN-5262. Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat.
2016-06-29 10:08:30 +05:30
Arun Suresh
99e5dd68d0
YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh)
2016-06-22 19:04:54 -07:00
Junping Du
d0162f2040
YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev.
2016-06-19 17:44:54 -07:00
Karthik Kambatla
20f2799938
YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha)
2016-06-17 22:24:42 -07:00
Wangda Tan
c77a1095dc
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:06:38 -07:00
Arun Suresh
5279af7cd4
YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh)
2016-06-10 22:33:42 -07:00
Rohith Sharma K S
e0f4620cc7
YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe.
2016-06-11 10:22:27 +05:30
Wangda Tan
620325e816
YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda)
2016-06-07 15:06:42 -07:00
Arun Suresh
3a154f75ed
YARN-4525. Fix bug in RLESparseResourceAllocation.getRangeOverlapping(). (Ishai Menache and Carlo Curino via asuresh)
2016-06-06 21:18:32 -07:00
Arun Suresh
7a9b7372a1
YARN-5185. StageAllocaterGreedyRLE: Fix NPE in corner case. (Carlo Curino via asuresh)
2016-06-06 21:06:52 -07:00
Ming Ma
4a1cedc010
MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma)
2016-06-06 14:30:51 -07:00
Arun Suresh
db54670e83
YARN-5165. Fix NoOvercommitPolicy to take advantage of RLE representation of plan. (Carlo Curino via asuresh)
2016-06-03 14:49:32 -07:00
Vinod Kumar Vavilapalli
f10ebc67f5
YARN-5098. Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time. Contributed by Jian He.
2016-06-03 13:00:07 -07:00
Jian He
097baaaeba
YARN-1815. Work preserving recovery of Unmanged AMs. Contributed by Subru Krishnan
2016-06-03 10:49:30 -07:00
Varun Vasudev
42f90ab885
YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 21:24:16 +05:30
Karthik Kambatla
04ded558b0
YARN-5035. FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on. (kasha)
2016-05-26 14:41:07 -07:00
Karthik Kambatla
4f513a4a8e
YARN-4866. FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy. (Yufei Gu via kasha)
2016-05-25 22:13:27 -07:00
Carlo Curino
013532a95e
YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino)
2016-05-25 16:55:49 -07:00
Karthik Kambatla
f979d779e1
YARN-4878. Expose scheduling policy and max running apps over JMX for Yarn queues. (Yufei Gu via kasha)
2016-05-24 10:54:11 -07:00
Naganarasimha
b4078bd17b
YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt
2016-05-24 08:06:53 +08:00
Karthik Kambatla
6d043aa4cf
YARN-4979. FSAppAttempt demand calculation considers demands at multiple locality levels different. (Zhihai Xu via kasha)
2016-05-23 14:29:28 -07:00
Jason Lowe
ac954486c5
YARN-5055. max apps per user can be larger than max per queue. Contributed by Eric Badger
2016-05-23 15:54:42 +00:00
Junping Du
22fcd819f0
YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron.
...
(cherry picked from commit 2703ec68712279494d67b0d76b7ac10e7a1628be)
2016-05-19 14:15:21 -07:00
Jian He
feb90ffcca
YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong
2016-05-19 13:01:36 -07:00
Arun Suresh
1597630681
YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh)
2016-05-18 18:46:00 -07:00
Jian He
fa3bc3405d
YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du
2016-05-17 12:52:19 -07:00
Arun Suresh
ccc93e7812
YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh)
2016-05-16 23:22:01 -07:00
Arun Suresh
f45bc5a83e
YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh)
2016-05-15 17:54:34 -07:00
Arun Suresh
f0ac18d001
YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh)
2016-05-13 13:38:36 -07:00
Jason Lowe
013000fbc2
YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger
2016-05-12 20:28:36 +00:00
Rohith Sharma K S
b7ac85259c
YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)
2016-05-12 15:17:49 +05:30
Karthik Kambatla
4b4e4c6ba8
YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha)
2016-05-11 17:36:21 -07:00
Junping Du
39f2bac38b
YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong.
2016-05-11 09:28:35 -07:00
Naganarasimha
2750fb900f
YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt
2016-05-08 22:49:25 +05:30
Yongjun Zhang
47c41e7ac7
YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang)
2016-05-06 21:50:09 -07:00
Jason Lowe
b2ed6ae731
YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena
2016-05-06 22:59:39 +00:00
Wangda Tan
23248f63aa
getApplicationReport call may raise NPE for removed queues. (Jian He via wangda)
2016-05-06 15:30:45 -07:00
Jian He
bb62e05925
YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan
2016-05-05 12:56:21 -07:00
Jason Lowe
d0da13229c
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
2016-05-05 14:07:54 +00:00
Jason Lowe
ed54f5f1ff
YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts
2016-05-03 20:03:41 +00:00
Jian He
dd80042c42
YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe
2016-04-28 21:27:25 -07:00
Jian He
4beff01354
YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode
2016-04-26 21:00:17 -07:00
Arun Suresh
341888a0aa
YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
2016-04-26 20:12:12 -07:00
Karthik Kambatla
4b1dcbbe0c
YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled
2016-04-26 05:10:09 -07:00
Arun Suresh
c282a08f38
YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
...
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
2016-04-24 22:38:33 -07:00
Wangda Tan
7cb3a3da96
YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda)
2016-04-22 11:40:32 -07:00
Karthik Kambatla
170c4fd4cd
YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha)
2016-04-20 23:58:12 -07:00
Wangda Tan
fdc46bfb37
YARN-4934. Reserved Resource for QueueMetrics needs to be handled correctly in few cases. (Sunil G via wangda)
2016-04-16 22:47:41 -07:00
Jason Lowe
69f3d428d5
YARN-4940. yarn node -list -all failed if RM start with decommissioned node. Contributed by sandflee
2016-04-15 20:36:45 +00:00
Jason Lowe
2a5da97f81
Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
...
This reverts commit 1cbcd4a491
.
2016-04-11 15:51:01 +00:00
Akira Ajisaka
1ff27f9d12
YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
2016-04-11 14:55:03 +09:00
Karthik Kambatla
ff95fd547b
YARN-4927. TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler. (Bibin A Chundatt via kasha)
2016-04-09 10:31:02 -07:00
Wangda Tan
ec06957941
YARN-3215. Respect labels in CapacityScheduler when computing headroom. (Naganarasimha G R via wangda)
2016-04-08 15:33:04 -07:00
Jian He
9cb0c963d2
YARN-4740. AM may not receive the container complete msg when it restarts. Contributed by Jun Gong
2016-04-08 11:20:35 -07:00
Jian He
93bacda08b
YARN-4769. Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI. Contributed by Varun Vasudev
2016-04-06 16:13:47 -07:00
Wangda Tan
21eb428448
YARN-4699. Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node. (Sunil G via wangda)
2016-04-05 16:24:11 -07:00
Jason Lowe
1cbcd4a491
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
2016-04-05 13:40:19 +00:00
Rohith Sharma K S
776b549e2a
YARN-4609. RM Nodes list page takes too much time to load. Contributed by Bibin A Chundatt
2016-04-05 14:47:25 +05:30
naganarasimha
5092c94195
YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt
2016-04-04 16:25:03 +05:30
Rohith Sharma K S
1e6f92977d
YARN-4607. Pagination support for AppAttempt page TotalOutstandingResource Requests table. Contributed by Bibin A Chundatt
2016-04-04 08:09:29 +05:30
Wangda Tan
12b11e2e68
YARN-4634. Scheduler UI/Metrics need to consider cases like non-queue label mappings. (Sunil G via wangda)
2016-03-31 14:35:18 -07:00