big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
11,573 Commits 2 Branches 2 Tags 264 MiB
7d6687fe76
Commit Graph

892 Commits

Author SHA1 Message Date
carlo curino
0e602fa3a1 YARN-3800. Reduce storage footprint for ReservationAllocation. Contributed by Anubhav Dhoot. 2015-07-09 16:51:59 -07:00
Jian He
c9dd2cada0 YARN-3892. Fixed NPE on RMStateStore#serviceStop when CapacityScheduler#serviceInit fails. Contributed by Bibin A Chundatt 2015-07-07 14:16:21 -07:00
Devaraj K
37d7395773 YARN-3875. FSSchedulerNode#reserveResource() doesn't print Application Id 
properly in log. Contributed by Bibin A Chundatt.
 2015-07-02 10:20:31 +05:30
Wangda Tan
0e4b06690f YARN-3508. Prevent processing preemption events on the main RM dispatcher. (Varun Saxena via wangda) 2015-07-01 17:32:22 -07:00
Devaraj K
80a68d6056 YARN-3830. AbstractYarnScheduler.createReleaseCache may try to clean a 
null attempt. Contributed by nijel.
 2015-07-01 19:03:44 +05:30
Devaraj K
b543d1a390 YARN-3859. LeafQueue doesn't print user properly for application add. 
Contributed by Varun Saxena.
 2015-06-28 10:04:50 +05:30
Devaraj K
57f1a01eda YARN-3826. Race condition in ResourceTrackerService leads to wrong 
diagnostics messages. Contributed by Chengbing Liu.
 2015-06-25 16:13:59 +05:30
rohithsharmaks
dd4b387d96 YARN-3790. usedResource from rootQueue metrics may get stale data for FS scheduler after recovering the container (Zhihai Xu via rohithsharmaks) 2015-06-24 23:00:14 +05:30
Jason Lowe
2a20dd9b61 YARN-3809. Failed to launch new attempts because ApplicationMasterLauncher's threads all hang. Contributed by Jun Gong 2015-06-24 16:23:48 +00:00
Xuan
5b5bb8dcdc YARN-3802. Two RMNodes for the same NodeId are used in RM sometimes 
after NM is reconnected. Contributed by zhihai xu
 2015-06-18 14:37:49 -07:00
Xuan
a826d432f9 YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena 2015-06-17 16:23:27 -07:00
Devaraj K
b039e69bb0 YARN-3789. Improve logs for LeafQueue#activateApplications(). Contributed 
by Bibin A Chundatt.
 2015-06-16 14:03:22 +05:30
Xuan
2b2465dfac YARN-3778. Fix Yarn resourcemanger CLI usage. Contributed by Brahma Reddy Battula 2015-06-08 15:43:03 -07:00
Jian He
960b8f19ca YARN-2716. Refactor ZKRMStateStore retry code with Apache Curator. Contributed by Karthik Kambatla 2015-06-08 14:50:58 -07:00
Devaraj K
c7ee6c151c YARN-3780. Should use equals when compare Resource in 
RMNodeImpl#ReconnectNodeTransition. Contributed by zhihai xu.
 2015-06-08 11:54:55 +05:30
Karthik Kambatla
bd69ea408f YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation and container reservation. (Zhihai Xu via kasha) 2015-06-07 11:37:52 -07:00
Xuan
3e000a919f YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong 2015-06-05 12:48:52 -07:00
Karthik Kambatla
75885852cc YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha) 2015-06-05 09:39:41 -07:00
Jian He
1970ca7cbc YARN-2392. Add more diags about app retry limits on AM failures. Contributed by Steve Loughran 2015-06-04 11:14:09 -07:00
Jian He
6ad4e59cfc YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. Contributed by Wangda Tan 2015-06-04 10:52:59 -07:00
Junping Du
d7e7f6aa03 YARN-41. The RM should handle the graceful shutdown of the NM. Contributed by Devaraj K. 2015-06-04 04:59:27 -07:00
Xuan
5766a04428 YARN-3749. We should make a copy of configuration when init 
MiniYARNCluster with multiple RMs. Contributed by Chun Chen
 2015-06-03 17:20:15 -07:00
Zhijie Shen
bc85959edd Revert "YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong." 
This reverts commit 4a9ec1a824.
 2015-06-03 14:15:56 -07:00
Karthik Kambatla
edb9cd0f7a YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) 2015-06-03 13:47:24 -07:00
Zhijie Shen
4a9ec1a824 YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong. 2015-05-30 21:05:36 -07:00
Karthik Kambatla
a8acdd65b3 YARN-3467. Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI. (Anubhav Dhoot via kasha) 2015-05-30 13:36:24 -07:00
Karthik Kambatla
3ae2a62501 YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) 2015-05-29 15:17:22 -07:00
Vinod Kumar Vavilapalli
9acd24fec4 Fixed more FilesSystemRMStateStore issues. Contributed by Vinod Kumar Vavilapalli. 2015-05-28 15:25:56 -07:00
Allen Wittenauer
d6e3164d4a YARN-2355. MAX_APP_ATTEMPTS_ENV may no longer be a useful env var for a container (Darrell Taylor via aw) 2015-05-27 16:40:56 -07:00
Wangda Tan
ec0a852a37 YARN-3647. RMWebServices api's should use updated api from CommonNodeLabelsManager to get NodeLabel object. (Sunil G via wangda) 2015-05-27 12:51:20 -07:00
Wangda Tan
cdbd66be11 YARN-3686. CapacityScheduler should trim default_node_label_expression. (Sunil G via wangda) 2015-05-26 15:58:47 -07:00
Jian He
10732d515f YARN-3632. Ordering policy should be allowed to reorder an application when demand changes. Contributed by Craig Welch 2015-05-26 12:00:51 -07:00
Tsuyoshi Ozawa
9a3d617b63 YARN-2336. Fair scheduler's REST API returns a missing '[' bracket JSON for deep queue tree. Contributed by Kenji Kikushima and Akira Ajisaka. 2015-05-26 19:07:40 +09:00
Jian He
446d51591e YARN-3707. RM Web UI queue filter doesn't work. Contributed by Wangda Tan 2015-05-22 15:58:22 -07:00
Karthik Kambatla
4513761869 YARN-3675. FairScheduler: RM quits when node removal races with continuous-scheduling on the same node. (Anubhav Dhoot via kasha) 2015-05-21 13:44:42 -07:00
Tsuyoshi Ozawa
7401e5b5e8 YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. Contributed by Vinod Kumar Vavilapalli. 2015-05-20 08:59:26 +09:00
Wangda Tan
563eb1ad2a YARN-3583. Support of NodeLabel object instead of plain String in YarnClient side. (Sunil G via wangda) 2015-05-19 16:54:38 -07:00
Wangda Tan
b37da52a1c YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String. (Naganarasimha G R via wangda) 2015-05-19 16:34:17 -07:00
Xuan
b0ad644083 YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang 2015-05-15 22:39:49 -07:00
Jason Lowe
f7e051c431 YARN-2421. RM still allocates containers to an app in the FINISHING state. Contributed by Chang Li 2015-05-15 22:09:30 +00:00
Vinod Kumar Vavilapalli
9a2a9553ee Fixing HDFS state-store. Contributed by Arun Suresh. 2015-05-14 16:13:51 -07:00
Junping Du
15ccd967ee YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in RMApps. Contributed by Xuan Gong. 2015-05-14 10:58:12 -07:00
Wangda Tan
0e85044e26 YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda) 2015-05-13 17:00:36 -07:00
Wangda Tan
7f19e7a254 YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda) 2015-05-13 13:43:17 -07:00
Wangda Tan
d6f6741296 YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) 2015-05-11 17:31:15 -07:00
Junping Du
7e543c27fa YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. projects. Contributed by Gabor Liptak. 2015-05-11 05:27:07 -07:00
Karthik Kambatla
a60f78e98e YARN-3395. FairScheduler: Trim whitespaces when using username for queuename. (Zhihai Xu via kasha) 2015-05-09 15:41:20 -07:00
Robert Kanter
5658998845 YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter) 2015-05-08 16:23:10 -07:00
Jian He
d0b73287c2 YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch. Contributed Zhihai Xu 2015-05-08 15:12:33 -07:00
Vinod Kumar Vavilapalli
f7a74d2491 YARN-3018. Unified the default value for the configuration property yarn.scheduler.capacity.node-locality-delay in code and default xml file. Contributed by Nijel SF. 2015-05-08 14:45:06 -07:00
Wangda Tan
f55d21d894 YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page. (Naganarasimha G R via wangda) 2015-05-08 12:04:50 -07:00
Thomas Graves
5d708a4725 YARN-3600. AM container link is broken (Naganarasimha G R via tgraves 2015-05-08 16:35:40 +00:00
Devaraj K
ef3d66d462 YARN-3358. Audit log not present while refreshing Service ACLs. 
Contributed by Varun Saxena.
 2015-05-08 12:14:10 +05:30
Devaraj K
a28cd02ba0 YARN-3592. Fix typos in RMNodeLabelsManager. Contributed by Sunil G. 2015-05-08 10:54:36 +05:30
Jian He
f489a4ec96 YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan 2015-05-07 17:35:41 -07:00
Jian He
b88700dcd0 YARN-3584. Fixed attempt diagnostics format shown on the UI. Contributed by nijel 2015-05-07 13:54:07 -07:00
Vinod Kumar Vavilapalli
4c7b9b6abe YARN-3385. Fixed a race-condition in ResourceManager's ZooKeeper based state-store to avoid crashing on duplicate deletes. Contributed by Zhihai Xu. 2015-05-06 17:51:17 -07:00
Jian He
0d3188fd25 YARN-3301. Fixed the format issue of the new RM attempt web page. Contributed by Xuan Gong 2015-05-06 13:45:00 -07:00
Xuan
b7dd3a4f04 YARN-2123. Progress bars in Web UI always at 100% due to non-US locale. 
Contributed by Akira AJISAKA
 2015-05-05 10:40:16 -07:00
Jason Lowe
9356cf8676 YARN-3552. RM Web UI shows -1 running containers for completed apps. Contributed by Rohith 2015-05-05 13:42:32 +00:00
Jian He
d701acc9c6 YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore. Contributed by Tsuyoshi Ozawa 2015-05-04 16:13:29 -07:00
Gera Shegalov
f8204e241d YARN-2893. AMLaucher: sporadic job failures due to EOFException in readTokenStorageStream. (Zhihai Xu via gera) 2015-05-01 18:18:55 -07:00
Zhijie Shen
7e8639fda4 YARN-3544. Got back AM logs link on the RM web UI for a completed app. Contributed by Xuan Gong. 2015-04-30 13:05:55 -07:00
tgraves
2e215484bd YARN-3517. RM web ui for dumping scheduler logs should be for admins only (Varun Vasudev via tgraves) 2015-04-29 21:25:42 +00:00
Karthik Kambatla
8f82970e0c YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) 2015-04-28 21:00:35 -07:00
Wangda Tan
db1b674b50 YARN-2740. Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled. (Naganarasimha G R via wangda) 2015-04-27 16:24:38 -07:00
Jian He
d497f6ea2b YARN-2498. Respect labels in preemption policy of capacity scheduler for inter-queue preemption. Contributed by Wangda Tan 2015-04-24 17:03:13 -07:00
Tsuyoshi Ozawa
4a3dabd94f YARN-3406. Display count of running containers in the RM's Web UI. Contributed by Ryu Kobayashi. 2015-04-25 07:17:11 +09:00
Jian He
d03dcb9635 YARN-3387. Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time. Contributed by Sandflee 2015-04-24 12:13:29 -07:00
Xuan
eee9facbba YARN-3511. Add errors and warnings page to ATS. Contributed by Varun Vasudev 2015-04-24 09:41:59 -07:00
Vinod Kumar Vavilapalli
f5fe35e297 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv) 2015-04-23 11:19:55 -07:00
Wangda Tan
395205444e YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) 2015-04-23 10:47:15 -07:00
tgraves
189a63a719 YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation 2015-04-23 14:39:25 +00:00
Steve Loughran
d9bcf99aef YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. (Xuan Gong via stevel) 2015-04-23 09:22:48 +01:00
Junping Du
fad9d7e85b New parameter of CLI for decommissioning node gracefully in RMAdmin CLI. Contributed by Devaraj K 2015-04-22 10:07:20 -07:00
Jian He
bdd90110e6 YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks 2015-04-21 20:06:20 -07:00
Wangda Tan
e71d0d87d9 YARN-3410. YARN admin should be able to remove individual application records from RMStateStore. (Rohith Sharmaks via wangda) 2015-04-21 17:51:22 -07:00
Tsuyoshi Ozawa
105afd5477 YARN-3495. Confusing log generated by FairScheduler. Contributed by Brahma Reddy Battula. 2015-04-22 05:47:59 +09:00
Wangda Tan
44872b76fc YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) 2015-04-20 17:12:32 -07:00
Jian He
497c86b485 YARN-3136. Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers. Contributed by Sunil G 2015-04-18 12:46:37 -07:00
Wangda Tan
f47a5763ac YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 2015-04-17 17:22:27 -07:00
Wangda Tan
f65eeb412d YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 2015-04-17 17:11:22 -07:00
Jian He
6779467ab6 YARN-3451. Display attempt start time and elapsed time on the web UI. Contributed by Rohith Sharmaks 2015-04-17 15:55:34 -07:00
Jian He
d573f09fb9 YARN-2696. Queue sorting in CapacityScheduler should consider node label. Contributed by Wangda Tan 2015-04-17 13:36:59 -07:00
Junping Du
1db355a875 YARN-1402. Update related Web UI and CLI with exposing client API to check log aggregation status. Contributed by Xuan Gong. 2015-04-17 13:18:59 -07:00
Jian He
bb6dde68f1 YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang 2015-04-16 19:43:37 -07:00
Akira Ajisaka
1fa8075794 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 2015-04-16 10:04:09 -05:00
Jian He
1b89a3e173 YARN-3354. Add node label expression in ContainerTokenIdentifier to support RM recovery. Contributed by Wangda Tan 2015-04-15 13:57:06 -07:00
Tsuyoshi Ozawa
e48cedc663 YARN-3326. Support RESTful API for getLabelsToNodes. Contributed by Naganarasimha G R. 2015-04-15 14:03:55 -05:00
Wangda Tan
5004e75332 YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. (Craig Welch via wangda) 2015-04-15 09:56:32 -07:00
Jian He
0fefda645b YARN-3361. CapacityScheduler side changes to support non-exclusive node labels. Contributed by Wangda Tan 2015-04-14 11:45:58 -07:00
Jian He
b46ee1e7a3 YARN-3266. RMContext#inactiveNodes should have NodeId as map key. Contributed by Chengbing Liu 2015-04-14 10:54:22 -07:00
Jian He
a1afbc48b5 YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. Contributed by Rohith Sharmaks 2015-04-13 14:07:17 -07:00
Jian He
944a16579f YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root queue metrics. Contributed by Rohit Agarwal 2015-04-13 10:53:34 -07:00
Karthik Kambatla
e516706b89 YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. (Jun Hong via kasha) 2015-04-10 11:20:34 -07:00
Jian He
577d755e4b YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 09:57:17 -07:00
Junping Du
92431c9617 YARN-1376. NM need to notify the log aggregation status to RM through Node heartbeat. Contributed by Xuan Gong. 2015-04-10 08:56:18 -07:00
Xuan
afa5d4715a YARN-3293. Track and display capacity scheduler health metrics in web 
UI. Contributed by Varun Vasudev
 2015-04-09 23:38:04 -07:00
Vinod Kumar Vavilapalli
9c5911294e YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 2015-04-09 13:08:53 -07:00
Wangda Tan
1885141e90 YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda) 2015-04-09 10:35:12 -07:00