big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,071 Commits 2 Branches 2 Tags 264 MiB
19f8e4f34d
Commit Graph

4224 Commits

Author SHA1 Message Date
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Eric E Payne
a7526ba9f7 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db302)
 2020-06-09 18:51:46 +00:00
Szilard Nemeth
ac307fe20d YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-05 12:40:57 +02:00
Szilard Nemeth
a266e32d82 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:50:43 +02:00
Szilard Nemeth
f65f64e8ae YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:32:06 +02:00
Prabhu Joseph
07b8963aa3 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.

(cherry picked from commit 6ce295b787)
 2020-06-01 18:24:38 +05:30
Jonathan Hung
7a323a45aa YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb66)
 2020-05-29 09:25:08 -07:00
Szilard Nemeth
94358dbc3c YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e)
 2020-05-27 13:07:17 +02:00
Szilard Nemeth
2c60bcb1c2 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit 9ce913a7bf.
Reverting because of missing Jira-ID in the commit message
 2020-05-27 13:07:01 +02:00
Szilard Nemeth
9ce913a7bf FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e)
 2020-05-22 14:48:53 +02:00
Jonathan Hung
564d3211f2 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c122)
 2020-05-12 10:49:10 -07:00
Szilard Nemeth
95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Ahmed Hussein
7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:28:04 -05:00
Ahmed Hussein
1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:24:01 -05:00
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
Szilard Nemeth
8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth
75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Szilard Nemeth
e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Akira Ajisaka
62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba)
 2020-04-28 05:45:37 +09:00
Szilard Nemeth
c0b7b38e22 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 
(cherry picked from commit f91e21ac10)
 2020-04-27 08:26:46 +02:00
Szilard Nemeth
7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth
61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka
497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961)
 2020-04-24 10:27:39 +09:00
Szilard Nemeth
1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Jonathan Hung
49ae9b2137 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:59 -07:00
Szilard Nemeth
f473473355 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 
(cherry picked from commit 55fcbcb5c2)
 2020-04-15 07:27:47 +02:00
Jonathan Hung
e1dd78143b YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
 2020-04-09 11:06:49 -07:00
Sunil G
94df6b513b YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 
(cherry picked from commit 6fb29d0f17)
 2020-04-09 18:52:52 +05:30
Wilfred Spiegelenburg
618a9208f7
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 
(cherry picked from commit 2214005c0f)
 2020-04-08 13:02:51 +10:00
Szilard Nemeth
7abc6221a3 YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 
(cherry picked from commit 45362a9f4c)
 2020-04-07 16:01:47 +02:00
Szilard Nemeth
3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth
9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth
348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Jonathan Hung
6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
Sunil G
cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Prabhu Joseph
d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue 
Contributed by Peter Bacsko. Reviewed by Manikandan R.
 2020-03-23 18:01:14 +05:30
Prabhu Joseph
f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node 
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
 2020-03-19 12:55:14 +05:30
Weiwei Yang
b9d825f178 YARN-9567. Add diagnostics for outstanding resource requests on app attempts page. Contributed by Tao Yang. 2020-03-13 14:39:36 -07:00
Bilwa
c83644deac YARN-10110. Adding RouterPolicyProvider for RM Federation. 
Contributed by Bilwa S T.
 2020-03-13 16:56:44 +05:30
Szilard Nemeth
5ead9c15ca YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-03-12 12:29:03 +01:00
Szilard Nemeth
38d87883b6 YARN-10193. FS-CS converter: fix incorrect capacity conversion. Contributed by Peter Bacsko 2020-03-12 12:18:37 +01:00
Sunil G
0fd8bf5f6b YARN-10191. FS-CS converter: call System.exit function call for every code path in main method. Contributed by Peter Bacsko. 2020-03-12 15:03:12 +05:30
Szilard Nemeth
cf9cf83a43 YARN-9354. Resources should be created with ResourceTypesTestHelper instead of TestUtils. Contributed by Andras Gyori 2020-03-10 16:44:48 +01:00
Szilard Nemeth
61f4cf3055 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-03-10 16:35:04 +01:00
Szilard Nemeth
9314ef947f YARN-10168. FS-CS Converter: tool doesn't handle min/max resource conversion correctly. Contributed by Peter Bacsko 2020-03-10 16:07:46 +01:00
Eric E Payne
ede05b19d1 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 2020-03-10 14:17:45 +00:00
Szilard Nemeth
44afe1154d YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-09 16:08:24 +01:00
bibinchundatt
3859fa76d0 YARN-6924. Metrics for Federation AMRMProxy. Contributed by Young Chen 2020-03-07 09:34:42 +05:30
Szilard Nemeth
ea0444851d YARN.10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-03-06 17:49:45 +01:00
Szilard Nemeth
004e955348 YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko 2020-03-05 09:56:42 +01:00
Eric Badger
2649f8b327 YARN-10173. Make pid file generation timeout configurable in case of reacquired 
container. Contributed by Adam Antal.
 2020-03-04 23:31:57 +00:00
Szilard Nemeth
c0d084247c YARN-10175. FS-CS converter: only convert placement rules if a cmd line switch is defined. Contributed by Peter Bacsko 2020-03-03 09:46:32 +01:00
Szilard Nemeth
e9eecedf69 YARN-10148. addendum: Fix method call parameter order of setAdminAndSubmitACL in TestCapacitySchedulerQueueACLs. Contributed by Kinga Marton 2020-03-02 16:25:14 +01:00
Inigo Goiri
a43510e21d YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 2020-02-27 13:18:30 -08:00
Inigo Goiri
b420ddeada YARN-10155. TestDelegationTokenRenewer.testTokenThreadTimeout fails in trunk. Contributed by Manikandan R. 2020-02-27 12:53:20 -08:00
Szilard Nemeth
10461e0193 YARN-10148. Add Unit test for queue ACL for both FS and CS. Contributed by Kinga Marton 2020-02-27 20:10:32 +01:00
Brahma Reddy Battula
3a9ccf7f6d YARN-10141.Interceptor in FederationInterceptorREST doesnt update on RM switchover. Contributed by D M Murali Krishna Reddy. 2020-02-26 23:24:00 +05:30
Szilard Nemeth
d68616b045 YARN-10130. FS-CS converter: Do not allow output dir to be the same as input dir. Contributed by Adam Antal 2020-02-25 21:30:04 +01:00
Szilard Nemeth
72fa9c7f1b YARN-10135. FS-CS converter tool: issue warning on dynamic auto-create mapping rules. Contributed by Peter Bacsko 2020-02-24 21:54:07 +01:00
Szilard Nemeth
34cf63c7d0 YARN-10157. FS-CS converter: initPropertyActions() is not called without rules file. Contributed by Peter Bacsko 2020-02-24 21:45:49 +01:00
Szilard Nemeth
3f1aad05f0 YARN-10147. FPGA plugin can't find the localized aocx file. Contributed by Peter Bacsko 2020-02-19 15:54:25 +01:00
Sunil G
6526f95bd2 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 2020-02-19 11:17:22 +05:30
Akira Ajisaka
d6d7f8d8c5
YARN-8374. Upgrade objenesis to 2.6 (#1798) 2020-02-19 09:50:37 +09:00
Brahma Reddy Battula
20add89718 YARN-10136. [Router] : Application metrics are hardcode as N/A in UI. Contributed by Bilwa S T. 2020-02-14 16:50:29 +05:30
Prabhu Joseph
e637797211 YARN-10127. Remove setting App Ordering Policy to ParentQueue in FSQueueConverter 
Contributed by Peter Bacsko.
 2020-02-11 22:01:58 +05:30
Sunil G
28f730b317 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 2020-02-09 21:14:53 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth
71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Jonathan Hung
314e2f9d2e YARN-10116. Expose diagnostics in RMAppManager summary 2020-02-04 17:44:05 -08:00
Giovanni Matteo Fumarola
bf8686f43f YARN-8982. [Router] Add locality policy. Contributed by Young Chen. 2020-01-30 16:59:36 -08:00
Szilard Nemeth
a7d72c523a YARN-10099. FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues. Contributed by Peter Bacsko 2020-01-30 16:03:38 +01:00
Akira Ajisaka
a5ef08b619
YARN-9743. [JDK11] TestTimelineWebServices.testContextFactory fails. (#1824) Contributed by Akira Ajisaka and Kinga Marton. 2020-01-30 14:10:31 +09:00
Prabhu Joseph
825db8fe2a YARN-10107. Fix GpuResourcePlugin#getNMResourceInfo to honor Auto Discovery Enabled 
Contributed by Szilard Nemeth.
 2020-01-29 13:30:00 +05:30
Eric Badger
e578e52aae YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 03:54:43 +00:00
Inigo Goiri
5abd0148eb YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-28 10:06:37 -08:00
Prabhu Joseph
1ab9c692fa YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change 
Contributed by Kinga Marton.
 2020-01-28 23:16:04 +05:30
Szilard Nemeth
94f060242e YARN-10104. FS-CS converter: dry run should work without output defined. Contributed by Peter Bacsko 2020-01-28 15:36:04 +01:00
Szilard Nemeth
ca29768035 YARN-10085. FS-CS converter: remove mixed ordering policy check. Contributed by Peter Bacsko 2020-01-28 15:22:48 +01:00
Szilard Nemeth
9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
Inigo Goiri
b4870bce3a Revert "YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R." 
This reverts commit 0696828a09.
 2020-01-21 17:45:17 -08:00
Inigo Goiri
0696828a09 YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-21 13:41:01 -08:00
Szilard Nemeth
581072a8f0 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-20 13:10:32 +01:00
Szilard Nemeth
8b3ee2f7e9 YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-20 09:54:22 +01:00
Prabhu Joseph
a0ff42d761 YARN-10070. Fix NPE if no queue mapping defined for proxy user when using application tag based placement. 
Contributed by Kinga Marton. Reviewed by Adam Antal.
 2020-01-16 12:55:27 +05:30
Akira Ajisaka
14c2c3d69d
YARN-9512. [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException. (#1806) 2020-01-16 12:21:50 +09:00
Szilard Nemeth
2aa065d98f YARN-10082. FS-CS converter: disable terminal placement rule checking. Contributed by Peter Bacsko 2020-01-15 12:42:13 +01:00
Szilard Nemeth
7c5cecc3b3 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-15 09:55:41 +01:00
Szilard Nemeth
1c51f36be7 YARN-9788. Queue Management API does not support parallel updates. Contributed by Prabhu Joseph 2020-01-14 12:26:03 +01:00
Szilard Nemeth
13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00
Szilard Nemeth
621c5eac38 YARN-9912. Capacity scheduler: support u:user2:%secondary_group queue mapping. Contributed by Manikandan R 2020-01-13 17:23:00 +01:00
Szilard Nemeth
b7ef8a333f YARN-9868. Validate %primary_group queue in CS queue manager. Contributed by Manikandan R 2020-01-13 17:15:09 +01:00
Szilard Nemeth
d842dfffa5 YARN-9866. u:user2:%primary_group is not working as expected. Contributed by Manikandan R 2020-01-12 14:04:15 +01:00
Szilard Nemeth
24e6a9e43a YARN-10067. Add dry-run feature to FS-CS converter tool. Contributed by Peter Bacsko 2020-01-12 13:48:39 +01:00
Eric Yang
2301b25899 YARN-10019. Improved container-executor exec() calls. 
Contributed by Peter Bacsko
 2020-01-10 19:04:04 -05:00
Eric E Payne
93233a7d6e YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 2020-01-09 17:18:44 +00:00
Eric E Payne
b1e07d27cc YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 19:26:01 +00:00
Eric E Payne
6899be5a17 YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 17:29:56 +00:00
Szilard Nemeth
dd2607e3ec YARN-10026. Pull out common code pieces from ATS v1.5 and v2. Contributed by Adam Antal 2020-01-06 17:16:11 +01:00
Szilard Nemeth
768ee22e9e YARN-10035. Add ability to filter the Cluster Applications API request by name. Contributed by Adam Antal 2020-01-06 16:26:33 +01:00
Prabhu Joseph
eca7e14c2f YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:13:57 +05:30
Prabhu Joseph
21ada4d1b0 Revert "YARN-10053. Use Shared Group Mapping Service in Placement Rules." 
This reverts commit 217b56ffdd.
 2020-01-02 14:12:43 +05:30
Prabhu Joseph
217b56ffdd YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:07:49 +05:30
Eric Badger
412035b47a YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 18:07:00 +00:00
Giovanni Matteo Fumarola
ef59ffd362 YARN-10038. [UI] Finish Time is not correctly parsed in the RM Apps page. Contributed by Inigo Goiri. 2019-12-19 11:37:17 -08:00
Jonathan Hung
7b93575b92 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 2019-12-18 13:18:11 -08:00
Jonathan Hung
fddc3d55c3 YARN-10039. Allow disabling app submission from REST endpoints 2019-12-18 10:48:05 -08:00
Eric Badger
f47dcf2d4c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 2019-12-17 17:11:23 +00:00
Szilard Nemeth
631dbbc6f2 YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal 2019-12-15 17:28:04 +01:00
prabhujoseph
d4bde134e3 YARN-7721. Fix TestContinuousScheduling fails sporadically with NPE. 
Contributed by Wilfred Spiegelenburg.
 2019-12-10 15:00:23 +05:30
Eric Badger
8dffd8dc89 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-09 01:25:10 +00:00
Eric Badger
ab2cc5ac92 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger" 
This reverts commit 76bb297db9.
 2019-12-09 01:25:10 +00:00
Jonathan Hung
6d970d8100 YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 2019-12-08 16:35:02 -08:00
Jonathan Hung
cd4779f317 Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit 92bce918dc.
 2019-12-08 16:34:46 -08:00
Jonathan Hung
92bce918dc YARN-10012. Guaranteed and max capacity queue metrics for custom resources 2019-12-08 15:52:17 -08:00
Eric Badger
76bb297db9 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-06 18:33:58 +00:00
prabhujoseph
e575df5ca6 YARN-10006. IOException used in place of YARNException in CapaitySheduler 
Contributed by Adam Antal. Reviewed by Szilard Nemeth.
 2019-12-06 12:51:28 +05:30
Szilard Nemeth
cf68857631 YARN-9938. Validate Parent Queue for QueueMapping contains dynamic group as parent queue. Contributed by Manikandan R 2019-12-05 21:04:42 +01:00
Szilard Nemeth
5cc6f945da YARN-9789. Disable Option for Write Ahead Logs of LogMutation. Contributed by Prabhu Joseph 2019-12-05 20:43:26 +01:00
Szilard Nemeth
4627dd6708 YARN-9780. SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call. Contributed by Prabhu Joseph 2019-12-05 20:38:37 +01:00
Szilard Nemeth
c71befaf8f YARN-9781. SchedConfCli to get current stored scheduler configuration. Contributed by Prabhu Joseph 2019-12-05 20:31:24 +01:00
Szilard Nemeth
520fe2c99b YARN-5106. Provide a builder interface for FairScheduler allocations for use in tests. Contributed by Adam Antal 2019-12-05 17:37:40 +01:00
Sunil G
682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
Wanqiang Ji
c48de9aa2d YARN-9958. Remove the invalid lock in ContainerExecutor (#1704) 2019-12-04 14:35:41 +09:00
Abhishek Modi
a2dadac790 YARN-9990. Testcase fails with Insufficient configured threads: required=16 < max=10. Contributed by Prabhu Joseph. 2019-11-29 11:54:33 +05:30
Szilard Nemeth
ef950b0863 YARN-9290. Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler. Contributed by Prabhu Joseph 2019-11-26 22:04:07 +01:00
Szilard Nemeth
828ab400ee YARN-9362. Code cleanup in TestNMLeveldbStateStoreService. Contributed by Denes Gerencser 2019-11-26 21:45:12 +01:00
prabhujoseph
aa7ab2719f YARN-9991. Fix Application Tag prefix to userid. Contributed by Szilard Nemeth. 2019-11-27 01:59:12 +05:30
Szilard Nemeth
8c9018d5c7 YARN-9899. Migration tool that help to generate CS config based on FS config [Phase 2]. Contributed by Peter Bacsko 2019-11-26 21:22:35 +01:00
Szilard Nemeth
448ffb12ec YARN-9937. addendum: Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2019-11-26 13:41:41 +01:00
Eric Badger
6c20512b68 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger" 
This reverts commit 289bbca870.
 2019-11-25 17:46:42 +00:00
HUAN-PING SU
f8e36e03b4 YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 2019-11-25 15:28:53 +09:00
Szilard Nemeth
4c1a1287bc YARN-9968. Public Localizer is exiting in NodeManager due to NullPointerException. Contributed by Tarun Parimi 2019-11-22 12:58:10 +01:00
Tao Yang
8afabe41e4 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:06:14 +08:00
Szilard Nemeth
dfdc6d6dd9 YARN-9886. Queue mapping based on userid passed through application tag. Contributed by Julia Kinga Marton 2019-11-19 21:14:15 +01:00
Prabhu Joseph
30b2096933 YARN-9986. Allow app owner to signal container via RMWebServices. Contributed by kyungwan nam. 2019-11-19 11:35:00 +00:00
Sunil G
ea68756c0c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:10:41 +05:30
Sunil G
215f2052fc YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:03:02 +05:30
Abhishek Modi
dc3f4fc2f4 YARN-9965. Fix NodeManager failing to start on subsequent times when Hdfs Auxillary Jar is set (addendum). Contributed by Prabhu Joseph. 2019-11-19 08:08:40 +05:30
Eric Badger
0e22e9ab83 YARN-9562. Add Java changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:56:31 +00:00
Eric Badger
289bbca870 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:54:16 +00:00
Szilard Nemeth
27642367ef YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-18 16:29:39 +01:00
Yufei Gu
b83b9ab418 YARN-9537. Add configuration to disable AM preemption. Contributed by ZhouKang 2019-11-12 09:40:16 -08:00
Abhishek Modi
fb512f5087 YARN-9697. Efficient allocation of Opportunistic containers. Contributed by Abhishek Modi. 2019-11-12 16:34:04 +05:30
Szilard Nemeth
30b93f914b YARN-9865. Capacity scheduler: add support for combined %user + %secondary_group mapping. Contributed by Manikandan R 2019-11-11 13:28:02 +01:00
Abhishek Modi
516377bfa6 YARN-9965. Fix NodeManager failing to start when Hdfs Auxillary Jar is set. Contributed by Prabhu Joseph. 2019-11-11 16:01:44 +05:30
Szilard Nemeth
31f172fd96 YARN-9677. Make FpgaDevice and GpuDevice classes more similar to each other. Contributed by kevin su 2019-11-09 14:04:14 +01:00
zhoukang
7d0adddf09 YARN-9689: Support proxy user for Router to support kerberos (#1654) 2019-11-04 17:13:29 -08:00
Sunil G
51e7d1b37e YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 2019-11-04 23:27:49 +05:30
Sunil G
d462308e04 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 2019-11-03 08:47:27 +05:30
Sunil G
9a2e43e29e YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph. 2019-10-31 00:34:27 +05:30
Szilard Nemeth
e3e7daa4f5 YARN-9889. [UI] Add Application Tag column to RM All Applications table. Contributed by Kinga Marton 2019-10-30 16:26:56 +01:00
Abhishek Modi
ed45c13f67 YARN-2442. ResourceManager JMX UI does not give HA State. Contributed by Rohith Sharma and Cyrus Jackson. 2019-10-29 15:54:55 +05:30
Eric Badger
eef34f2d87 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 2019-10-25 17:13:04 +00:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Zhankun Tang
fd84ca5161 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:05:00 +08:00
Eric E Payne
83d148074f YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 2019-10-21 20:30:50 +00:00
Eric Yang
3f7756dc6c YARN-9922. Fixed YARN javadoc errors from YARN-9699. 
Contributed by Peter Bacsko
 2019-10-21 16:16:49 -04:00
Eric Yang
155864da00 YARN-9875. Improve fair scheduler configuration store on HDFS. 
Contributed by Prabhu Joseph
 2019-10-18 20:30:11 -04:00
Eric Yang
72b1bed998 YARN-9884. Make container-executor mount logic modular 
Contributed by Eric Badger
 2019-10-18 19:10:32 -04:00
Mate Szalay-Beko
6d92aa7c30 HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 (#1656). Contributed by Norbert Kalmár, Mate Szalay-Beko 
* HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5

- Add a static initializer for the unit tests using ZooKeeper to enable
the four-letter-words diagnostic telnet commands. (this is an interface
that become disabled by default, so to keep the ZooKeeper 3.4.x behavior
we enabled it for the tests)
- Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt
records. The new ZooKeeper seems to respond quicker during the fail-over
tests than the ZooKeeper, so we made sure to catch all the relevant records
by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().

Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>
 2019-10-18 13:26:20 -07:00
Szilard Nemeth
f0699a7406 YARN-9841. Capacity scheduler: add support for combined %user + %primary_group mapping. Contributed by Manikandan R 2019-10-18 11:19:49 +02:00
Sunil G
3990ffa083 YARN-9909. Offline format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-10-17 06:43:06 +05:30
Eric E Payne
a5034c7988 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 2019-10-16 21:10:08 +00:00
Sunil G
090f73a9aa YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 2019-10-16 18:14:31 +05:30
Szilard Nemeth
a8ef03e961 YARN-9840. Capacity scheduler: add support for Secondary Group rule mapping. Contributed by Manikandan R 2019-10-15 17:22:52 +02:00
Szilard Nemeth
5cc7873a47 YARN-9699. Migration tool that help to generate CS config based on FS config [Phase 1]. Contributed by Peter Bacsko 2019-10-14 17:19:39 +02:00
Szilard Nemeth
ec86f42e40 YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-11 14:01:19 +02:00
Eric Yang
31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN 
Contributed by Prabhu Joseph and Shane Kumpf
 2019-10-10 19:02:02 -04:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Sunil G
4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Abhishek Modi
61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
bibinchundatt
425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Sunil G
137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Eric Badger
18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Jonathan Hung
c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Jonathan Hung
3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Sunil G
c474e24c0b YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 2019-09-17 19:41:38 +05:30
Abhishek Modi
da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
HUAN-PING SU
4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
 2019-09-13 10:36:45 -07:00
Abhishek Modi
44850f6784 YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph. 2019-09-12 12:50:10 +05:30
Abhishek Modi
3b06f0bf9e YARN-9819. Make TestOpportunisticContainerAllocatorAMService more resilient. Contribued by Abhishek Modi 2019-09-12 08:09:57 +05:30
Eric Badger
56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Jonathan Hung
f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang
10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. 
Contributed by Prabhu Joseph
 2019-09-10 17:04:39 -04:00
Abhishek Modi
60af8793b4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 14:26:00 +05:30
Jonathan Hung
387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00
Jonathan Hung
43e389b980 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:07:11 -07:00
Abhishek Modi
162af6f554 YARN-9784. org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky. Contributed by Julia Kinga Marton. 2019-09-07 18:08:04 +05:30
Wangda Tan
34b82e6da0 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
 2019-09-06 19:18:13 -07:00
Tao Yang
73575701ab YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:52:39 +08:00
Jonathan Hung
b71a7f195c YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 11:00:07 -07:00
Jonathan Hung
e7d44e48f7 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:17:07 -07:00
Jonathan Hung
0ccf4b0fe1 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 2019-09-05 14:01:42 -07:00