big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,841 Commits 2 Branches 2 Tags 264 MiB
28c533a582
Commit Graph

4738 Commits

Author SHA1 Message Date
slfan1989
454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
slfan1989
d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
slfan1989
9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
slfan1989
48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
slfan1989
5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989
1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989
647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989
82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Szilard Nemeth
0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
Ashutosh Gupta
9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 (#4937) 
Contributed by PJ Fanning
 2022-10-07 15:44:01 +01:00
slfan1989
b31b3ea0f6
YARN-11187. Remove WhiteBox in yarn module. (#4463) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-06 23:13:33 +09:00
slfan1989
a708ff96f1
YARN-11324. Fix some PBImpl classes to avoid NPE. (#4961) 2022-10-04 14:30:20 -04:00
slfan1989
874a004347
YARN-11318. Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils. (#4939) 2022-10-04 09:18:01 -07:00
slfan1989
22bd5e3b53
YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. (#4904) 2022-10-04 09:17:00 -07:00
Riya Khandelwal
07581f1ab2
YARN-6169 message on empty configuration file improved (#4952) 2022-10-03 23:31:06 -04:00
slfan1989
42d883937d
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924) 2022-09-27 13:31:52 -07:00
slfan1989
bfd6415827
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846) 2022-09-27 13:28:52 -07:00
slfan1989
aeba204fa2
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897) 2022-09-26 15:46:06 -07:00
Ashutosh Gupta
603e9bd745
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:30:58 +09:00
Ashutosh Gupta
fd0415c44a
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:26:06 +09:00
slfan1989
e526f48fa4
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858) 2022-09-22 16:40:47 -07:00
Ashutosh Gupta
917aef75fc
YARN-11255. Support loading alternative docker client config from system environment (#4884) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-09-21 16:54:02 +05:30
slfan1989
4c5a7cc6fc
YARN-11307. Fix Yarn Router Broken Link. (#4905) 2022-09-20 23:10:09 -04:00
slfan1989
fd687bb4c4
YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241. (#4893). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:06:24 +05:30
slfan1989
f52b900a5f
YARN-11283. Fix Typo of NodeManager amrmproxy. (#4899) 2022-09-19 13:16:25 -07:00
slfan1989
342c4856b8
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. (#4852) 2022-09-19 13:14:55 -07:00
slfan1989
88545a875d
YARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-11269. (#4878) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:08:16 +09:00
Ashutosh Gupta
55d8a91b2c
YARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:05:24 +09:00
slfan1989
3ce353395b
YARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843) 2022-09-12 12:33:21 -07:00
Ashutosh Gupta
65a027b112
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-12 23:32:20 +09:00
Ashutosh Gupta
21bae31d58
YARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (#4772) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-11 03:31:28 +09:00
slfan1989
b2760520c3
YARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 22:57:06 +05:30
slfan1989
e76ffbf102
YARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863) 2022-09-09 10:39:00 -07:00
9uapaw
5b85af87f0 YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta. 2022-09-09 14:38:41 +02:00
slfan1989
29c4d8d8f7
YARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870) 2022-09-08 11:42:36 -07:00
slfan1989
7bf95d7949
YARN-11289. [Federation] Improve NM FederationInterceptor removeAppFromRegistry. (#4836) 2022-09-02 10:41:31 -07:00
slfan1989
1965708d49
YARN-11273. Federation StateStore: Support storage/retrieval of Reservations With SQL. (#4817) 2022-09-02 10:39:58 -07:00
slfan1989
b266f852d7
YARN-11284. [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop (#4814) 2022-09-02 10:28:38 -07:00
slfan1989
3a96de7756
YARN-6667. Handle containerId duplicate without failing the heartbeat in Federation Interceptor. (#4810) 2022-09-02 10:25:26 -07:00
slfan1989
37e213c3fc
YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation. (#4764) 2022-09-01 16:35:20 -07:00
slfan1989
33edbed54c
YARN-11272. Federation StateStore: Support storage/retrieval of Reservations With Zk. (#4781) 2022-08-31 10:15:15 -07:00
slfan1989
8a47ed6f84
YARN-11287. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-10793 (#4828) 
Co-authored-by: slfan1989 <louj1988@@>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-30 20:41:04 +09:00
Samrat
2c05015716
YARN-11196. NUMA support in DefaultContainerExecutor (#4742) 2022-08-30 10:39:41 +05:30
slfan1989
c60a900583
YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager. (#4792) 2022-08-27 10:17:00 -07:00
slfan1989
0075ef15c2
YARN-8482. [Router] Add cache for fast answers to getApps. (#4769) 2022-08-27 10:14:55 -07:00
slfan1989
4031b0774e
YARN-11253. Add Configuration to delegationToken RemoverScanInterval. (#4751) 2022-08-27 10:02:59 -07:00
slfan1989
f8b9dd911c
YARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router. (#4757) 2022-08-26 16:01:17 -07:00
slfan1989
eda4bb5dcd
YARN-11250. Capture the Performance Metrics of ZookeeperFederationStateStore. (#4738) 2022-08-22 14:09:20 -07:00
Ashutosh Gupta
c294a414b9
YARN-9425. Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService (#4731). Contributed by groot and Shen Yinjie. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-22 03:40:00 +05:30
Clara Fang
c870171182
YARN-11254. hadoop-minikdc dependency duplicated in hadoop-yarn-server-nodemanager (#4755) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:09:42 +09:00
Ashutosh Gupta
b253b3be9f
YARN-11269. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timeline-pluginstorage (#4771) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 06:52:23 +09:00
slfan1989
f75c58a1ca
YARN-11252. Yarn Federation Router Supports Update / Delete Reservation in MemoryStore. (#4741) 2022-08-18 21:13:43 -07:00
slfan1989
cd72f7e042
YARN-11224. [Federation] Add getAppQueue, updateAppQueue REST APIs for Router. (#4747) 2022-08-17 13:13:07 -07:00
Ashutosh Gupta
f02ff1afe2
YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING (#4721) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-16 19:07:05 +09:00
slfan1989
eff3b8c59a
YARN-10885. Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache. (#4701) 2022-08-15 13:46:40 -07:00
slfan1989
ab88e4b65d
YARN-11223. [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router. (#4733) 2022-08-14 19:22:16 -07:00
slfan1989
d383cc4525
YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. (#4711) 2022-08-13 10:37:20 -07:00
zhengchenyu
9f6bbc90a8
YARN-11148. In federation and security mode, nm recover may fail. (#4308) 2022-08-13 10:33:16 -07:00
slfan1989
6ca2d3f848
YARN-6539. Create SecureLogin inside Router. (#4712) 2022-08-11 13:25:51 -07:00
slfan1989
133e8aabf0
YARN-11227. [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router. (#4715) 2022-08-10 14:53:46 -07:00
slfan1989
ffa9ed93a4
YARN-6572. Refactoring Router services to use common util classes for pipeline creations. (#4594) 2022-08-09 14:44:29 -07:00
Ashutosh Gupta
1cda2dcb6e
YARN-10793. Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice (#4603) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-07 16:15:47 +09:00
slfan1989
52c2d99889
YARN-11228. [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router. (#4695) 2022-08-06 09:36:26 -07:00
slfan1989
6f7c4c74ea
YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster (#4656) 2022-08-04 10:16:08 -07:00
slfan1989
c5ec727435
YARN-11230. [Federation] Add getContainer, signalToContainer REST APIs for Router. (#4689) 2022-08-03 11:21:48 -07:00
slfan1989
6463f86f83
YARN-11029. Refactor AMRMProxy Service code and Added Some Metrics. (#4650) 2022-08-03 09:38:00 -07:00
slfan1989
c5eba323bc
YARN-6972. Adding RM ClusterId in AppInfo. (#4673) 2022-08-03 09:35:40 -07:00
slfan1989
1f0a71a92b
YARN-8973. [Router] Add missing methods in RMWebProtocol. (#4664) 2022-08-02 14:07:09 -07:00
slfan1989
57da4bb0a1
YARN-11220. [Federation] Add getLabelsToNodes, getClusterNodeLabels, getLabelsOnNode REST APIs for Router (#4657) 2022-08-02 12:09:55 -07:00
slfan1989
2680f17eb4
YARN-11180. Refactor some code of getNewApplication, submitApplication etc. (#4618) 2022-07-29 08:23:11 -07:00
slfan1989
e994635a95
YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router. (#4614) 2022-07-28 11:53:04 -07:00
9uapaw
bf570bd4ac YARN-11063. Support auto queue creation template wildcards for arbitrary queue depths. Contributed by Bence Kosztolnik. 2022-07-28 17:32:20 +02:00
slfan1989
bf8782d0ac
YARN-10883. [Router] Router Audit Log Add Client IP Address. (#4426) 2022-07-25 11:55:40 -07:00
slfan1989
edeb99548a
YARN-11161. Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation (#4610) 2022-07-25 10:05:45 -07:00
slfan1989
63db1a85e3
YARN-11203. Fix typo in hadoop-yarn-server-router module. (#4510). Contributed by fanshilun. 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-23 20:28:45 +05:30
slfan1989
5c84cb81ba
YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs (#4543) 2022-07-22 17:06:38 -07:00
slfan1989
838020ce3b
YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation (#4540) 2022-07-21 11:57:24 -07:00
Szilard Nemeth
f4b635c4dc YARN-11211. QueueMetrics leaks Configuration objects when validation API is called multiple times. Contributed by Andras Gyori 2022-07-21 14:20:34 +02:00
PJ Fanning
34e548cb62
HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api (#4547) 
This downgrades jackson from the version switched to in
    HADOOP-18033 (2.13.0), to Jackson 2.12.7.
    This removes the dependency on javax.ws.rs-api,
    so avoiding runtime problems with applications using
    jersey-core v1 and/or jsr311-api.
    
    The 2.12.7 release still contains the fix for CVE-2020-36518.
    
    Contributed by PJ Fanning
 2022-07-17 21:37:54 +05:30
Samrat
84ce592a85
YARN-11198. clean up numa resources from statestore (#4546) 
* YARN-11198. clean up numa resources from levelDB

Co-authored-by: Deb <dbsamrat@3c22fba1b03f.ant.amazon.com>
 2022-07-14 11:07:48 +05:30
slfan1989
161b1fac2e
YARN-11169. Support moveApplicationAcrossQueues, getQueueInfo API's for Federation. (#4464) 2022-07-05 11:24:29 -07:00
Ashutosh Gupta
151bb31c47
YARN-9403.GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table (#4516) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-02 21:59:28 +05:30
Ashutosh Gupta
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping (#4515) 
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-02 21:58:56 +05:30
Tamas Domok
3cad632709 YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok. 
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
 2022-07-01 10:50:48 +02:00
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Ashutosh Gupta
a177232ebc
YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down (#4492) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-28 09:32:07 +05:30
Ashutosh Gupta
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation (#4487) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-23 21:59:27 +05:30
slfan1989
0af4bb3b42
YARN-11192. TestRouterWebServicesREST failing after YARN-9827. (#4484). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-23 13:21:36 +05:30
Szilard Nemeth
3a66348fda YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori 2022-06-20 15:03:58 +02:00
slfan1989
62e4476102
YARN-10122. Support signalToContainer API for Federation. (#4421) 2022-06-17 16:38:36 -07:00
zhengchenyu
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. (#4408) 
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.

Contributed by zhengchenyu
 2022-06-17 19:49:36 +01:00
Benjamin Teke
020201cb65 Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu. 2022-06-17 19:28:32 +02:00
slfan1989
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation (#4396) 2022-06-08 17:34:43 -07:00
slfan1989
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation (#4341) 2022-06-02 12:54:29 -07:00
Ashutosh Gupta
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project (#4381). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 23:17:11 +05:30
Ashutosh Gupta
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-26 15:51:55 +09:00
slfan1989
678768035d
YARN-11137. Improve log message in FederationClientInterceptor (#4336) 2022-05-24 17:21:04 -07:00
Owen O'Malley
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 2022-05-24 14:07:19 +09:00
slfan1989
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation (#4317) 2022-05-20 13:19:42 -07:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
9uapaw
0e6a6d1880 YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation 2022-05-18 18:22:42 +02:00
9uapaw
54cd0174c0 YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe 2022-05-18 18:16:34 +02:00
slfan1989
6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274) 2022-05-15 09:16:06 -07:00
Ashutosh Gupta
6985f9aabe
YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:41:06 +09:00
Jian Chen
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110) 
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:11:42 +09:00
Szilard Nemeth
eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Szilard Nemeth
5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
Szilard Nemeth
9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth
b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989
09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
章锡平
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. (#4175) 2022-04-15 17:50:49 +09:00
章锡平
3b46aae977 YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1 2022-04-13 15:57:18 +02:00
Junfan Zhang
e044a46f97 YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang 2022-03-31 12:39:56 +02:00
PJ Fanning
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-26 20:31:16 +09:00
Tamas Domok
da09d68056 YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok 2022-03-25 21:53:18 +01:00
9uapaw
921267ca31 YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang. 2022-03-23 08:57:00 +01:00
9uapaw
2beb7296fb YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang. 2022-03-22 13:14:43 +01:00
9uapaw
c3124a3f6e YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke. 2022-03-21 14:55:49 +01:00
Szilard Nemeth
e2701e227f YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang. 2022-03-21 14:37:10 +01:00
Szilard Nemeth
4537b34e1c YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang. 2022-03-21 14:18:27 +01:00
Szilard Nemeth
ed65aa2324 YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori 2022-03-10 22:22:58 +01:00
Szilard Nemeth
481da19494 YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke 2022-03-10 22:15:35 +01:00
9uapaw
383b73417d YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-03-10 13:11:19 +01:00
9uapaw
9539ff108a YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth 2022-03-10 08:40:43 +01:00
Szilard Nemeth
db8ae4b654 YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori 2022-03-09 19:36:09 +01:00
Szilard Nemeth
2ece95064b YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori 2022-03-09 19:29:53 +01:00
Viraj Jasani
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing (#4027) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-08 21:32:51 +09:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
9uapaw
851a4dc2d9 YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth 2022-03-07 09:01:18 +01:00
Szilard Nemeth
379baa5eb6 YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-03-03 16:44:12 +01:00
Szilard Nemeth
356d337d1e YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok 2022-03-02 16:19:17 +01:00
9uapaw
9e475aede6 YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke 2022-03-02 11:16:24 +01:00
Szilard Nemeth
365375412f YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori 2022-02-22 12:26:35 +01:00
Szilard Nemeth
0463498adc YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori 2022-02-22 11:55:34 +01:00
Szilard Nemeth
7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth
48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741) 
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-02-01 12:18:54 +05:30
minni31
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse (#3766) 2022-01-25 09:25:18 -08:00
Andrew Chung
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container (#3779) 2022-01-24 08:03:36 -08:00
Gera Shegalov
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-17 16:20:48 +09:00
Andras Gyori
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
minni31
3bf014d871
YARN-8859. Add audit logs for router service (#3607) 
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-01-03 10:38:59 +05:30
Cheng Pan
4db27acf06
YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 11:08:04 +09:00
Ashutosh Gupta
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 17:14:51 +09:00
Eric Payne
e2d6fd075d YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:05:39 +00:00
Viraj Jasani
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled (#3802) 2021-12-20 18:02:57 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Szilard Nemeth
07141426e0
YARN-11050 (#3805) 2021-12-18 11:52:03 +08:00
Szilard Nemeth
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class (#3800). Contributed by Szilard Nemeth 2021-12-17 00:18:14 +01:00
Szilard Nemeth
aec9cdb467 YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok 2021-12-16 23:39:18 +01:00
Szilard Nemeth
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API (#3799). Contributed by Szilard Nemeth 2021-12-16 15:53:08 +01:00
Szilard Nemeth
6ea1685abd YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke 2021-12-16 00:01:09 +01:00
Viraj Jasani
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (#3796) 2021-12-15 16:47:51 +08:00
Szilard Nemeth
843f66f4dc Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke 2021-12-14 22:00:46 +01:00
Szilard Nemeth
364d38f00d YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke 2021-12-14 14:57:08 +01:00
Szilard Nemeth
a5bcf4c792 YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke 2021-12-13 21:57:46 +01:00
Szilard Nemeth
898055e204 YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke 2021-12-13 16:12:55 +01:00
Viraj Jasani
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown (#3768) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-13 14:03:32 +09:00
Szilard Nemeth
f544ef4a51 YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok 2021-12-10 15:09:53 +01:00
Szilard Nemeth
a0d8cde133 YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs 2021-12-09 17:51:44 +01:00
Andrew Chung
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy (#3717) 2021-12-08 10:02:10 -08:00
Szilard Nemeth
66a08e933a YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok 2021-12-08 16:41:35 +01:00
Szilard Nemeth
19430118c6 YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2021-12-08 16:03:42 +01:00
Viraj Jasani
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3749) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 16:52:22 +09:00
Szilard Nemeth
126079612c YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori 2021-12-07 15:55:16 +01:00
Szilard Nemeth
7284d23476 YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke 2021-12-07 15:43:55 +01:00
Szilard Nemeth
60440a836a YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok 2021-12-07 15:26:07 +01:00
Szilard Nemeth
d75f2db2f9 YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2021-12-07 08:42:57 +01:00
Ashutosh Gupta
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-06 19:45:44 +09:00
Andras Gyori
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars (#3710) 
Contributed by Andras Gyori
 2021-12-03 16:44:58 +00:00
Viraj Jasani
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile (#3722) 2021-12-02 10:40:30 +08:00
Sushmasree-28
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files (#3735) 
Contributed by  sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
 2021-11-30 00:00:25 +05:30
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 (#3642) 2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources (#3646) 2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules (#3687) 2021-11-23 13:36:22 +09:00
Szilard Nemeth
7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
Szilard Nemeth
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration (#3570). Contributed by Szilard Nemeth 2021-10-28 15:19:32 +02:00
Szilard Nemeth
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue (#3551) contributed by Szilard Nemeth 2021-10-27 19:03:45 +02:00
Szilard Nemeth
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler (#3581) Contributed by Szilard Nemeth 2021-10-27 17:13:49 +02:00
Jack
9cfd8d0a83 YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth 2021-10-23 14:47:09 +02:00
9uapaw
32ecaed9c3 YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori 2021-10-22 17:32:33 +02:00
Adam Antal
23772d946b YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal 2021-10-22 16:33:03 +02:00
Ahmed Hussein
d286994009 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-20 22:18:36 +00:00
Szilard Nemeth
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth 2021-10-20 15:52:37 +02:00