big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,629 Commits 2 Branches 2 Tags 264 MiB
e0938b4c2a
Commit Graph

4677 Commits

Author SHA1 Message Date
Szilard Nemeth
815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth
29f2230cb6 YARN-5607. Document TestContainerResourceUsage#waitForContainerCompletion. Contributed by Susheel Gupta 2023-01-25 15:13:24 +01:00
slfan1989
3f767a61b1
YARN-8900. [Follow Up] Fix FederationInterceptorREST#invokeConcurrent Inaccurate Order of Subclusters. (#5260) 2023-01-19 17:13:55 -08:00
slfan1989
442a5fb285
YARN-11320. [Federation] Add getSchedulerInfo REST APIs for Router. (#5217) 2023-01-17 09:36:19 -08:00
slfan1989
168fa07801
YARN-11409. Fix Typo of ResourceManager#webapp moudle. (#5285) 2023-01-13 11:53:13 +08:00
slfan1989
4520448327
YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. (#5193) 2023-01-12 15:04:09 -08:00
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-27 15:58:53 -08:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:15:36 +01:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:20:53 +01:00
slfan1989
6172c3192d
YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. (#5056) 2022-12-14 14:37:56 -08:00
slfan1989
63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. (#5131) 2022-12-14 09:09:38 -08:00
slfan1989
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. (#5182) 2022-12-10 15:03:49 -08:00
K0K0V0K
ee7d1787cd
YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-08 09:52:19 -08:00
slfan1989
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. (#5146) 2022-12-06 08:17:05 -08:00
Gautham B A
dadd3d9138
YARN-11386. Fix issue with classpath resolution (#5183) 
* This PR ensures that all the special notations such as
  <CPS> are resolved before getting added to classpath.
 2022-12-06 16:32:26 +05:30
slfan1989
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. (#5104) 2022-12-01 13:20:21 -08:00
Szilard Nemeth
5440c75c4a YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs 2022-12-01 15:11:58 +01:00
slfan1989
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. (#5130) 2022-11-22 14:38:24 -08:00
Szilard Nemeth
3c37a01654 YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta 2022-11-22 13:37:55 +01:00
Ashutosh Gupta
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common (#4717) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:40:02 +09:00
slfan1989
eccd2d0492
YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs. (#5057) 2022-11-16 18:00:38 -08:00
Szilard Nemeth
22c9f28f4d YARN-10005. Code improvements in MutableCSConfigurationProvider. Contributed by Peter Szucs 2022-11-12 18:35:49 +01:00
slfan1989
b398a7b003
YARN-11367. [Federation] Fix DefaultRequestInterceptorREST Client NPE. (#5100) 2022-11-09 10:25:10 -08:00
slfan1989
845cf8bc28
YARN-11368. [Federation] Improve Yarn Router's Federation Page style. (#5105) 2022-11-07 10:13:23 -08:00
slfan1989
5d6ab15860
YARN-11354. [Federation] Add Yarn Router's NodeLabel Web Page. (#5073) 2022-11-04 14:39:57 -07:00
slfan1989
b90dfdff3f
YARN-11366. Improve equals, hashCode(), toString() methods of the Federation Base Object. (#5096) 2022-11-03 21:33:53 -07:00
Ashutosh Gupta
e62ba16a02
HADOOP-18484. Upgrade hsqldb to v2.7.1 to mitigate CVE-2022-41853 (#4991) 2022-11-02 08:41:27 +01:00
Ashutosh Gupta
83acb55981
YARN-11364. Docker Container to accept docker Image name with sha256 digest (#5092) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:44:35 -07:00
Ashutosh Gupta
69225ae5b9
YARN-11363. Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests (#5091) 
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:02:06 -07:00
Ashutosh Gupta
cbe02c2e77
YARN-11264. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-tests (#4776) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:33:57 +09:00
slfan1989
070a2d4880
YARN-11332. [Federation] Improve FederationClientInterceptor#ThreadPool thread pool configuration. (#4982) 2022-10-28 15:39:04 -07:00
slfan1989
b1cd88c598
YARN-11229. [Federation] Add checkUserAccessToQueue REST APIs for Router. (#4929) 2022-10-28 15:37:35 -07:00
Chris Nauroth
bfb84cd7f6
YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 2022-10-28 11:07:01 -07:00
slfan1989
ba77530ff4
YARN-11357. Fix FederationClientInterceptor#submitApplication Can't Update SubClusterId (#5055) 2022-10-26 16:42:22 -07:00
slfan1989
454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
slfan1989
d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
slfan1989
9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
slfan1989
48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
slfan1989
5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989
1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989
647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989
82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Szilard Nemeth
0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
Ashutosh Gupta
9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 (#4937) 
Contributed by PJ Fanning
 2022-10-07 15:44:01 +01:00
slfan1989
b31b3ea0f6
YARN-11187. Remove WhiteBox in yarn module. (#4463) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-06 23:13:33 +09:00
slfan1989
a708ff96f1
YARN-11324. Fix some PBImpl classes to avoid NPE. (#4961) 2022-10-04 14:30:20 -04:00
slfan1989
874a004347
YARN-11318. Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils. (#4939) 2022-10-04 09:18:01 -07:00
slfan1989
22bd5e3b53
YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. (#4904) 2022-10-04 09:17:00 -07:00
Riya Khandelwal
07581f1ab2
YARN-6169 message on empty configuration file improved (#4952) 2022-10-03 23:31:06 -04:00
slfan1989
42d883937d
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924) 2022-09-27 13:31:52 -07:00
slfan1989
bfd6415827
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846) 2022-09-27 13:28:52 -07:00
slfan1989
aeba204fa2
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897) 2022-09-26 15:46:06 -07:00
Ashutosh Gupta
603e9bd745
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:30:58 +09:00
Ashutosh Gupta
fd0415c44a
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:26:06 +09:00
slfan1989
e526f48fa4
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858) 2022-09-22 16:40:47 -07:00
Ashutosh Gupta
917aef75fc
YARN-11255. Support loading alternative docker client config from system environment (#4884) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-09-21 16:54:02 +05:30
slfan1989
4c5a7cc6fc
YARN-11307. Fix Yarn Router Broken Link. (#4905) 2022-09-20 23:10:09 -04:00
slfan1989
fd687bb4c4
YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241. (#4893). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:06:24 +05:30
slfan1989
f52b900a5f
YARN-11283. Fix Typo of NodeManager amrmproxy. (#4899) 2022-09-19 13:16:25 -07:00
slfan1989
342c4856b8
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. (#4852) 2022-09-19 13:14:55 -07:00
slfan1989
88545a875d
YARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-11269. (#4878) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:08:16 +09:00
Ashutosh Gupta
55d8a91b2c
YARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:05:24 +09:00
slfan1989
3ce353395b
YARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843) 2022-09-12 12:33:21 -07:00
Ashutosh Gupta
65a027b112
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-12 23:32:20 +09:00
Ashutosh Gupta
21bae31d58
YARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (#4772) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-11 03:31:28 +09:00
slfan1989
b2760520c3
YARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 22:57:06 +05:30
slfan1989
e76ffbf102
YARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863) 2022-09-09 10:39:00 -07:00
9uapaw
5b85af87f0 YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta. 2022-09-09 14:38:41 +02:00
slfan1989
29c4d8d8f7
YARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870) 2022-09-08 11:42:36 -07:00
slfan1989
7bf95d7949
YARN-11289. [Federation] Improve NM FederationInterceptor removeAppFromRegistry. (#4836) 2022-09-02 10:41:31 -07:00
slfan1989
1965708d49
YARN-11273. Federation StateStore: Support storage/retrieval of Reservations With SQL. (#4817) 2022-09-02 10:39:58 -07:00
slfan1989
b266f852d7
YARN-11284. [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop (#4814) 2022-09-02 10:28:38 -07:00
slfan1989
3a96de7756
YARN-6667. Handle containerId duplicate without failing the heartbeat in Federation Interceptor. (#4810) 2022-09-02 10:25:26 -07:00
slfan1989
37e213c3fc
YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation. (#4764) 2022-09-01 16:35:20 -07:00
slfan1989
33edbed54c
YARN-11272. Federation StateStore: Support storage/retrieval of Reservations With Zk. (#4781) 2022-08-31 10:15:15 -07:00
slfan1989
8a47ed6f84
YARN-11287. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-10793 (#4828) 
Co-authored-by: slfan1989 <louj1988@@>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-30 20:41:04 +09:00
Samrat
2c05015716
YARN-11196. NUMA support in DefaultContainerExecutor (#4742) 2022-08-30 10:39:41 +05:30
slfan1989
c60a900583
YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager. (#4792) 2022-08-27 10:17:00 -07:00
slfan1989
0075ef15c2
YARN-8482. [Router] Add cache for fast answers to getApps. (#4769) 2022-08-27 10:14:55 -07:00
slfan1989
4031b0774e
YARN-11253. Add Configuration to delegationToken RemoverScanInterval. (#4751) 2022-08-27 10:02:59 -07:00
slfan1989
f8b9dd911c
YARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router. (#4757) 2022-08-26 16:01:17 -07:00
slfan1989
eda4bb5dcd
YARN-11250. Capture the Performance Metrics of ZookeeperFederationStateStore. (#4738) 2022-08-22 14:09:20 -07:00
Ashutosh Gupta
c294a414b9
YARN-9425. Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService (#4731). Contributed by groot and Shen Yinjie. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-22 03:40:00 +05:30
Clara Fang
c870171182
YARN-11254. hadoop-minikdc dependency duplicated in hadoop-yarn-server-nodemanager (#4755) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:09:42 +09:00
Ashutosh Gupta
b253b3be9f
YARN-11269. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timeline-pluginstorage (#4771) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 06:52:23 +09:00
slfan1989
f75c58a1ca
YARN-11252. Yarn Federation Router Supports Update / Delete Reservation in MemoryStore. (#4741) 2022-08-18 21:13:43 -07:00
slfan1989
cd72f7e042
YARN-11224. [Federation] Add getAppQueue, updateAppQueue REST APIs for Router. (#4747) 2022-08-17 13:13:07 -07:00
Ashutosh Gupta
f02ff1afe2
YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING (#4721) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-16 19:07:05 +09:00
slfan1989
eff3b8c59a
YARN-10885. Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache. (#4701) 2022-08-15 13:46:40 -07:00