big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,666 Commits 2 Branches 2 Tags 264 MiB
9f1e23cc67
Commit Graph

6681 Commits

Author SHA1 Message Date
slfan1989
b406060c6b
YARN-8972. [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size. (#5382) 2023-03-08 13:29:30 -08:00
Varun Saxena
2a0dc2ab2f YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary
 2023-03-05 21:25:16 +05:30
slfan1989
bcc51ce2c5
YARN-11375. [Federation] Support refreshAdminAcls、refreshServiceAcls API's for Federation. (#5312) 2023-02-28 14:44:00 -08:00
slfan1989
8798b94ee1
YARN-11221. [Federation] Add replaceLabelsOnNodes, replaceLabelsOnNode REST APIs for Router. (#5302) 2023-02-27 09:34:39 -08:00
slfan1989
25ebd0b8b1
YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router. (#5328) 2023-02-24 10:52:57 -08:00
slfan1989
27a54955f9
YARN-5604. [Federation] Add versioning for FederationStateStore. (#5394) 2023-02-24 10:51:19 -08:00
slfan1989
2e997d818d
YARN-11370. [Federation] Refactor MemoryFederationStateStore code. (#5126) 2023-02-22 12:37:35 -08:00
susheel-gupta
49b8ac19f2
YARN-11408. Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method (#5278) 
* added a check of autoQueueCreationV2Disabled

Change-Id: If1e36c5969d270c1b81a4bbd2e883fa819c81f20

* added check of AutoCreateChildQueueDisabled

Change-Id: Ia011b4393ccd8d4d419a2e46b06a5237d050851c

* removed auto-create-child-queue-enabled check and emit

Change-Id: I7a154124519ecbd81379b46a238707c16db1e82a
 2023-02-22 09:46:42 +01:00
slfan1989
4e6e2f318c
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-02-21 14:39:32 +00:00
slfan1989
7e486038ea
YARN-11439. Fix Typo of hadoop-yarn-ui README.md. (#5405) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:14:01 +08:00
Steve Loughran
10e7ca481c
YARN-11441. Revert YARN-10495. 
This reverts commit 7d3c8ef606.
 2023-02-17 15:05:06 +00:00
slfan1989
a5f48eacca
YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) 2023-02-15 14:40:34 -08:00
slfan1989
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Viraj Jasani
4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989
af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
slfan1989
a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
jokercurry
dad73b76c0
YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc (#5309) 
Co-authored-by: smallzhongfeng <982458633@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-04 10:29:19 +08:00
Masatake Iwasaki
a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
slfan1989
468135a4d9
YARN-11218. [Federation] Add getActivities, getBulkActivities REST APIs for Router. (#5284) 2023-01-26 11:14:05 -08:00
Szilard Nemeth
cf1b3711cb YARN-10965. Centralize queue resource calculation based on CapacityVectors. Contributed by Andras Gyori 2023-01-26 19:45:54 +01:00
Szilard Nemeth
815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth
29f2230cb6 YARN-5607. Document TestContainerResourceUsage#waitForContainerCompletion. Contributed by Susheel Gupta 2023-01-25 15:13:24 +01:00
slfan1989
3f767a61b1
YARN-8900. [Follow Up] Fix FederationInterceptorREST#invokeConcurrent Inaccurate Order of Subclusters. (#5260) 2023-01-19 17:13:55 -08:00
slfan1989
442a5fb285
YARN-11320. [Federation] Add getSchedulerInfo REST APIs for Router. (#5217) 2023-01-17 09:36:19 -08:00
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
slfan1989
168fa07801
YARN-11409. Fix Typo of ResourceManager#webapp moudle. (#5285) 2023-01-13 11:53:13 +08:00
slfan1989
4520448327
YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. (#5193) 2023-01-12 15:04:09 -08:00
slfan1989
3d21cff263
YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. (#5289) 
* YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

* YARN-11413. Fix CheckStyle.

* YARN-11413. Fix CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2023-01-12 14:29:05 +01:00
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
Riya Khandelwal
dd49077aed
YARN-6412 aux-services classpath not documented (#5242) 2023-01-09 15:30:24 +01:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-27 15:58:53 -08:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:15:36 +01:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:20:53 +01:00
slfan1989
6172c3192d
YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. (#5056) 2022-12-14 14:37:56 -08:00
slfan1989
63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. (#5131) 2022-12-14 09:09:38 -08:00
slfan1989
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. (#5182) 2022-12-10 15:03:49 -08:00
K0K0V0K
ee7d1787cd
YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-08 09:52:19 -08:00
slfan1989
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. (#5146) 2022-12-06 08:17:05 -08:00
Gautham B A
dadd3d9138
YARN-11386. Fix issue with classpath resolution (#5183) 
* This PR ensures that all the special notations such as
  <CPS> are resolved before getting added to classpath.
 2022-12-06 16:32:26 +05:30
slfan1989
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5179) 2022-12-02 10:56:17 -08:00
slfan1989
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. (#5104) 2022-12-01 13:20:21 -08:00
Szilard Nemeth
5440c75c4a YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs 2022-12-01 15:11:58 +01:00
slfan1989
f93167e678
YARN-11380. Fix hadoop-yarn-api module Java Doc Errors. (#5152). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-28 18:54:32 +05:30
slfan1989
1ddc9091f6
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5153). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-26 21:01:07 +05:30
slfan1989
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. (#5130) 2022-11-22 14:38:24 -08:00
Szilard Nemeth
3c37a01654 YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta 2022-11-22 13:37:55 +01:00
Ashutosh Gupta
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common (#4717) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:40:02 +09:00
slfan1989
eccd2d0492
YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs. (#5057) 2022-11-16 18:00:38 -08:00
Szilard Nemeth
22c9f28f4d YARN-10005. Code improvements in MutableCSConfigurationProvider. Contributed by Peter Szucs 2022-11-12 18:35:49 +01:00
slfan1989
b398a7b003
YARN-11367. [Federation] Fix DefaultRequestInterceptorREST Client NPE. (#5100) 2022-11-09 10:25:10 -08:00
slfan1989
845cf8bc28
YARN-11368. [Federation] Improve Yarn Router's Federation Page style. (#5105) 2022-11-07 10:13:23 -08:00
slfan1989
5d6ab15860
YARN-11354. [Federation] Add Yarn Router's NodeLabel Web Page. (#5073) 2022-11-04 14:39:57 -07:00
Steve Vaughan
2ba982a061
MAPREDUCE-7386. Maven parallel builds (skipping tests) fail (#4415) 
Contributed by Steve Vaughan Jr
 2022-11-04 11:50:43 +00:00
slfan1989
b90dfdff3f
YARN-11366. Improve equals, hashCode(), toString() methods of the Federation Base Object. (#5096) 2022-11-03 21:33:53 -07:00
Ashutosh Gupta
e62ba16a02
HADOOP-18484. Upgrade hsqldb to v2.7.1 to mitigate CVE-2022-41853 (#4991) 2022-11-02 08:41:27 +01:00
Ashutosh Gupta
83acb55981
YARN-11364. Docker Container to accept docker Image name with sha256 digest (#5092) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:44:35 -07:00
Ashutosh Gupta
69225ae5b9
YARN-11363. Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests (#5091) 
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:02:06 -07:00
Gautham B A
b1f418f802
YARN-11365. Fix NM class not found on Windows (#5093) 2022-10-31 09:19:07 -07:00
Ashutosh Gupta
cbe02c2e77
YARN-11264. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-tests (#4776) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:33:57 +09:00
Ashutosh Gupta
c096803387
YARN-11339. Upgrade Junit 4 to 5 in hadoop-yarn-services-api (#4995) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 03:07:16 +09:00
Ashutosh Gupta
2aae7ffe08
YARN-11337. Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo (#4993) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 02:25:53 +09:00
slfan1989
070a2d4880
YARN-11332. [Federation] Improve FederationClientInterceptor#ThreadPool thread pool configuration. (#4982) 2022-10-28 15:39:04 -07:00
slfan1989
b1cd88c598
YARN-11229. [Federation] Add checkUserAccessToQueue REST APIs for Router. (#4929) 2022-10-28 15:37:35 -07:00
Chris Nauroth
bfb84cd7f6
YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 2022-10-28 11:07:01 -07:00
slfan1989
ba77530ff4
YARN-11357. Fix FederationClientInterceptor#submitApplication Can't Update SubClusterId (#5055) 2022-10-26 16:42:22 -07:00
Bence Kosztolnik
562b693374 YARN-11356. Upgrade DataTables to 1.11.5 to fix CVEs. Contributed by Bence Kosztolnik. 2022-10-26 22:29:01 +02:00
Ashutosh Gupta
e6edbf1b4b
YARN-11338. Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher (#4994) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:21:12 +09:00
Ashutosh Gupta
21b7790866
YARN-11336. Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp (#4992) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 16:52:31 +09:00
slfan1989
454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
slfan1989
d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
slfan1989
9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
slfan1989
48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
slfan1989
ee886cacd7
YARN-11247. Remove unused classes introduced by YARN-9615. (#4720) 2022-10-18 09:40:39 -04:00
slfan1989
5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989
1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989
647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989
82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Szilard Nemeth
0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
Ashutosh Gupta
9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 (#4937) 
Contributed by PJ Fanning
 2022-10-07 15:44:01 +01:00
slfan1989
b31b3ea0f6
YARN-11187. Remove WhiteBox in yarn module. (#4463) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-06 23:13:33 +09:00
slfan1989
1a9faf123d
YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md. (#4927) 2022-10-05 13:25:20 -07:00
slfan1989
a708ff96f1
YARN-11324. Fix some PBImpl classes to avoid NPE. (#4961) 2022-10-04 14:30:20 -04:00
slfan1989
874a004347
YARN-11318. Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils. (#4939) 2022-10-04 09:18:01 -07:00
slfan1989
22bd5e3b53
YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. (#4904) 2022-10-04 09:17:00 -07:00
Riya Khandelwal
07581f1ab2
YARN-6169 message on empty configuration file improved (#4952) 2022-10-03 23:31:06 -04:00
slfan1989
42d883937d
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924) 2022-09-27 13:31:52 -07:00
slfan1989
bfd6415827
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846) 2022-09-27 13:28:52 -07:00
slfan1989
0e65f4cc04
YARN-11316. Fix Yarn federation.md table format. (#4935) 2022-09-26 15:47:08 -07:00