big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,081 Commits 2 Branches 2 Tags 264 MiB
60033fd581
Commit Graph

6606 Commits

Author SHA1 Message Date
slfan1989
2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). (#5625) 2023-05-12 18:12:05 -07:00
susheel-gupta
0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade (#5649) 2023-05-12 14:20:26 +02:00
Steve Loughran
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
slfan1989
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00
cxzl25
be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist (#5565) 2023-05-10 15:16:33 +08:00
slfan1989
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. (#5363) 2023-05-09 16:17:23 -07:00
slfan1989
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. (#5403) 2023-05-08 10:13:09 -07:00
slfan1989
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616) 2023-05-05 13:43:40 -07:00
slfan1989
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. (#5609) 2023-05-05 13:41:38 -07:00
Szilard Nemeth
c7699d3dcd YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta 2023-05-04 22:16:18 -04:00
Peter Szucs
bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum (#5614) 2023-05-04 12:27:25 +02:00
slfan1989
476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. (#5587) 2023-05-03 09:21:56 -07:00
slfan1989
c1d10f3872
YARN-9049. Add application submit data to state store. (#5606) 2023-05-03 09:19:54 -07:00
slfan1989
668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. (#5596) 2023-05-02 13:46:06 -07:00
slfan1989
87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. (#5598) 2023-05-01 16:01:50 -07:00
Hexiaoqiao
70c0aa342e
YARN-11482. Fix bug of DRF comparision DominantResourceFairnessComparator2 in fair scheduler. (#5607). Contributed by Xiaoqiao He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2023-04-29 11:18:42 +08:00
slfan1989
5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. (#5570) 
Co-authored-by: slfan1989 <louj1988@@>
 2023-04-28 14:11:13 -07:00
Riya Khandelwal
60a7e8acaa
YARN-11459 Changed label called "max resource" on UIv1 and UIv2 (#5527) 2023-04-27 15:25:25 +02:00
yl09099
245fde17d7
YARN-11474.The yarn queue list is displayed on the CLI (#5577) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-27 18:05:48 +08:00
slfan1989
55eebcf277
YARN-11378. [Federation] Support checkForDecommissioningNodes、refreshClusterMaxPriority API's for Federation. (#5551) 2023-04-25 14:12:38 -07:00
slfan1989
a716459cdf
YARN-11437. [Federation] SQLFederationStateStore Support Version. (#5589) 2023-04-25 14:09:45 -07:00
Ashutosh Gupta
964c1902c8
YARN-11463. Node Labels root directory creation doesn't have a retry logic (#5562) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2023-04-21 14:03:22 +02:00
rdingankar
5119d0c72f
HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot (#5556) 2023-04-18 10:47:37 -07:00
slfan1989
a258f1f235
YARN-11326. [Federation] Add RM FederationStateStoreService Metrics. (#4963) 2023-04-18 09:13:08 -07:00
slfan1989
635521db4c
YARN-11438. [Federation] ZookeeperFederationStateStore Support Version. (#5537) 2023-04-18 09:05:52 -07:00
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
yl09099
2c4d6bf33d
YARN-11465. Improved YarnClient Log Format (#5550) 
Co-authored-by: yl09099 <shaq376260428@163.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-17 09:27:52 +08:00
slfan1989
0bcdea7912
YARN-11239. Optimize FederationClientInterceptor audit log. (#5127) 2023-04-14 13:09:18 -07:00
slfan1989
06f9bdffa6
YARN-10846. Add dispatcher metrics to NM. (#4687) 2023-04-12 09:53:20 -07:00
slfan1989
dd6d0ac510
YARN-11462. Fix Typo of hadoop-yarn-common. (#5539) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-12 11:08:23 +08:00
slfan1989
bffa49a64f
YARN-11377. [Federation] Support addToClusterNodeLabels、removeFromClusterNodeLabels、replaceLabelsOnNode API's for Federation. (#5525) 2023-04-11 09:47:58 -07:00
slfan1989
69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
slfan1989
eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00
slfan1989
5bc8f25327
YARN-11446. [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. (#5476) 2023-03-28 09:33:19 -07:00
slfan1989
aa602381c5
YARN-11426. Improve YARN NodeLabel Memory Display. (#5335) 
YARN-11426. Improve YARN NodeLabel Memory Display.
Co-authored-by: slfan1989 <louj1988@@>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-03-28 22:48:46 +08:00
slfan1989
926993cb73
YARN-11376. [Federation] Support updateNodeResource、refreshNodesResources API's for Federation. (#5496) 2023-03-27 09:27:21 -07:00
Tamas Domok
69748aae32
YARN-11461. fix NPE in determineMissingParents (auto queue creation / CS). (#5506) 
Change-Id: Iaaaf43a545588eaff8a0a20f6f3c27258a45f390
 2023-03-24 09:38:53 +01:00
slfan1989
fa723ae839
YARN-11445. [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. (#5472) 2023-03-17 11:59:45 -07:00
Viraj Jasani
cf4a678ce9
HADOOP-18649. CLA and CRLA appenders to be replaced with RFA (#5448) 2023-03-16 00:46:17 +08:00
slfan1989
b406060c6b
YARN-8972. [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size. (#5382) 2023-03-08 13:29:30 -08:00
Varun Saxena
2a0dc2ab2f YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary
 2023-03-05 21:25:16 +05:30
slfan1989
bcc51ce2c5
YARN-11375. [Federation] Support refreshAdminAcls、refreshServiceAcls API's for Federation. (#5312) 2023-02-28 14:44:00 -08:00
slfan1989
8798b94ee1
YARN-11221. [Federation] Add replaceLabelsOnNodes, replaceLabelsOnNode REST APIs for Router. (#5302) 2023-02-27 09:34:39 -08:00
slfan1989
25ebd0b8b1
YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router. (#5328) 2023-02-24 10:52:57 -08:00
slfan1989
27a54955f9
YARN-5604. [Federation] Add versioning for FederationStateStore. (#5394) 2023-02-24 10:51:19 -08:00
slfan1989
2e997d818d
YARN-11370. [Federation] Refactor MemoryFederationStateStore code. (#5126) 2023-02-22 12:37:35 -08:00
susheel-gupta
49b8ac19f2
YARN-11408. Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method (#5278) 
* added a check of autoQueueCreationV2Disabled

Change-Id: If1e36c5969d270c1b81a4bbd2e883fa819c81f20

* added check of AutoCreateChildQueueDisabled

Change-Id: Ia011b4393ccd8d4d419a2e46b06a5237d050851c

* removed auto-create-child-queue-enabled check and emit

Change-Id: I7a154124519ecbd81379b46a238707c16db1e82a
 2023-02-22 09:46:42 +01:00
slfan1989
4e6e2f318c
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-02-21 14:39:32 +00:00
slfan1989
7e486038ea
YARN-11439. Fix Typo of hadoop-yarn-ui README.md. (#5405) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:14:01 +08:00
Steve Loughran
10e7ca481c
YARN-11441. Revert YARN-10495. 
This reverts commit 7d3c8ef606.
 2023-02-17 15:05:06 +00:00
slfan1989
a5f48eacca
YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) 2023-02-15 14:40:34 -08:00
slfan1989
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Viraj Jasani
4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989
af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
slfan1989
a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
jokercurry
dad73b76c0
YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc (#5309) 
Co-authored-by: smallzhongfeng <982458633@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-04 10:29:19 +08:00
Masatake Iwasaki
a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
slfan1989
468135a4d9
YARN-11218. [Federation] Add getActivities, getBulkActivities REST APIs for Router. (#5284) 2023-01-26 11:14:05 -08:00
Szilard Nemeth
cf1b3711cb YARN-10965. Centralize queue resource calculation based on CapacityVectors. Contributed by Andras Gyori 2023-01-26 19:45:54 +01:00
Szilard Nemeth
815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth
29f2230cb6 YARN-5607. Document TestContainerResourceUsage#waitForContainerCompletion. Contributed by Susheel Gupta 2023-01-25 15:13:24 +01:00
slfan1989
3f767a61b1
YARN-8900. [Follow Up] Fix FederationInterceptorREST#invokeConcurrent Inaccurate Order of Subclusters. (#5260) 2023-01-19 17:13:55 -08:00
slfan1989
442a5fb285
YARN-11320. [Federation] Add getSchedulerInfo REST APIs for Router. (#5217) 2023-01-17 09:36:19 -08:00
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
slfan1989
168fa07801
YARN-11409. Fix Typo of ResourceManager#webapp moudle. (#5285) 2023-01-13 11:53:13 +08:00
slfan1989
4520448327
YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. (#5193) 2023-01-12 15:04:09 -08:00
slfan1989
3d21cff263
YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. (#5289) 
* YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

* YARN-11413. Fix CheckStyle.

* YARN-11413. Fix CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2023-01-12 14:29:05 +01:00
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
Riya Khandelwal
dd49077aed
YARN-6412 aux-services classpath not documented (#5242) 2023-01-09 15:30:24 +01:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-27 15:58:53 -08:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:15:36 +01:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:20:53 +01:00
slfan1989
6172c3192d
YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. (#5056) 2022-12-14 14:37:56 -08:00
slfan1989
63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. (#5131) 2022-12-14 09:09:38 -08:00
slfan1989
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. (#5182) 2022-12-10 15:03:49 -08:00
K0K0V0K
ee7d1787cd
YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-08 09:52:19 -08:00
slfan1989
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. (#5146) 2022-12-06 08:17:05 -08:00
Gautham B A
dadd3d9138
YARN-11386. Fix issue with classpath resolution (#5183) 
* This PR ensures that all the special notations such as
  <CPS> are resolved before getting added to classpath.
 2022-12-06 16:32:26 +05:30
slfan1989
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5179) 2022-12-02 10:56:17 -08:00
slfan1989
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. (#5104) 2022-12-01 13:20:21 -08:00
Szilard Nemeth
5440c75c4a YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs 2022-12-01 15:11:58 +01:00
slfan1989
f93167e678
YARN-11380. Fix hadoop-yarn-api module Java Doc Errors. (#5152). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-28 18:54:32 +05:30
slfan1989
1ddc9091f6
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5153). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-11-26 21:01:07 +05:30
slfan1989
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. (#5130) 2022-11-22 14:38:24 -08:00
Szilard Nemeth
3c37a01654 YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta 2022-11-22 13:37:55 +01:00
Ashutosh Gupta
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common (#4717) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-11-21 14:40:02 +09:00
slfan1989
eccd2d0492
YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs. (#5057) 2022-11-16 18:00:38 -08:00
Szilard Nemeth
22c9f28f4d YARN-10005. Code improvements in MutableCSConfigurationProvider. Contributed by Peter Szucs 2022-11-12 18:35:49 +01:00
slfan1989
b398a7b003
YARN-11367. [Federation] Fix DefaultRequestInterceptorREST Client NPE. (#5100) 2022-11-09 10:25:10 -08:00
slfan1989
845cf8bc28
YARN-11368. [Federation] Improve Yarn Router's Federation Page style. (#5105) 2022-11-07 10:13:23 -08:00
slfan1989
5d6ab15860
YARN-11354. [Federation] Add Yarn Router's NodeLabel Web Page. (#5073) 2022-11-04 14:39:57 -07:00
slfan1989
b90dfdff3f
YARN-11366. Improve equals, hashCode(), toString() methods of the Federation Base Object. (#5096) 2022-11-03 21:33:53 -07:00
Ashutosh Gupta
e62ba16a02
HADOOP-18484. Upgrade hsqldb to v2.7.1 to mitigate CVE-2022-41853 (#4991) 2022-11-02 08:41:27 +01:00
Ashutosh Gupta
83acb55981
YARN-11364. Docker Container to accept docker Image name with sha256 digest (#5092) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:44:35 -07:00
Ashutosh Gupta
69225ae5b9
YARN-11363. Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests (#5091) 
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-11-01 14:02:06 -07:00
Gautham B A
b1f418f802
YARN-11365. Fix NM class not found on Windows (#5093) 2022-10-31 09:19:07 -07:00
Ashutosh Gupta
cbe02c2e77
YARN-11264. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-tests (#4776) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:33:57 +09:00
Ashutosh Gupta
c096803387
YARN-11339. Upgrade Junit 4 to 5 in hadoop-yarn-services-api (#4995) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 03:07:16 +09:00
Ashutosh Gupta
2aae7ffe08
YARN-11337. Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo (#4993) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 02:25:53 +09:00
slfan1989
070a2d4880
YARN-11332. [Federation] Improve FederationClientInterceptor#ThreadPool thread pool configuration. (#4982) 2022-10-28 15:39:04 -07:00
slfan1989
b1cd88c598
YARN-11229. [Federation] Add checkUserAccessToQueue REST APIs for Router. (#4929) 2022-10-28 15:37:35 -07:00
Chris Nauroth
bfb84cd7f6
YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 2022-10-28 11:07:01 -07:00
slfan1989
ba77530ff4
YARN-11357. Fix FederationClientInterceptor#submitApplication Can't Update SubClusterId (#5055) 2022-10-26 16:42:22 -07:00
Bence Kosztolnik
562b693374 YARN-11356. Upgrade DataTables to 1.11.5 to fix CVEs. Contributed by Bence Kosztolnik. 2022-10-26 22:29:01 +02:00
Ashutosh Gupta
e6edbf1b4b
YARN-11338. Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher (#4994) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:21:12 +09:00
Ashutosh Gupta
21b7790866
YARN-11336. Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp (#4992) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 16:52:31 +09:00
slfan1989
454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
slfan1989
d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
slfan1989
9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
slfan1989
48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
slfan1989
ee886cacd7
YARN-11247. Remove unused classes introduced by YARN-9615. (#4720) 2022-10-18 09:40:39 -04:00
slfan1989
5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989
1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989
647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989
82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Szilard Nemeth
0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
Ashutosh Gupta
9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 (#4937) 
Contributed by PJ Fanning
 2022-10-07 15:44:01 +01:00
slfan1989
b31b3ea0f6
YARN-11187. Remove WhiteBox in yarn module. (#4463) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-06 23:13:33 +09:00
slfan1989
1a9faf123d
YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md. (#4927) 2022-10-05 13:25:20 -07:00
slfan1989
a708ff96f1
YARN-11324. Fix some PBImpl classes to avoid NPE. (#4961) 2022-10-04 14:30:20 -04:00
slfan1989
874a004347
YARN-11318. Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils. (#4939) 2022-10-04 09:18:01 -07:00
slfan1989
22bd5e3b53
YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. (#4904) 2022-10-04 09:17:00 -07:00
Riya Khandelwal
07581f1ab2
YARN-6169 message on empty configuration file improved (#4952) 2022-10-03 23:31:06 -04:00
slfan1989
42d883937d
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924) 2022-09-27 13:31:52 -07:00
slfan1989
bfd6415827
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846) 2022-09-27 13:28:52 -07:00
slfan1989
0e65f4cc04
YARN-11316. Fix Yarn federation.md table format. (#4935) 2022-09-26 15:47:08 -07:00
slfan1989
aeba204fa2
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897) 2022-09-26 15:46:06 -07:00
Ashutosh Gupta
603e9bd745
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:30:58 +09:00
Ashutosh Gupta
fd0415c44a
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:26:06 +09:00
slfan1989
e526f48fa4
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858) 2022-09-22 16:40:47 -07:00
Ashutosh Gupta
917aef75fc
YARN-11255. Support loading alternative docker client config from system environment (#4884) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-09-21 16:54:02 +05:30
slfan1989
4c5a7cc6fc
YARN-11307. Fix Yarn Router Broken Link. (#4905) 2022-09-20 23:10:09 -04:00
slfan1989
fd687bb4c4
YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241. (#4893). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:06:24 +05:30
Ashutosh Gupta
a134628d1b
YARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895) 2022-09-20 09:50:03 -07:00
slfan1989
f52b900a5f
YARN-11283. Fix Typo of NodeManager amrmproxy. (#4899) 2022-09-19 13:16:25 -07:00
slfan1989
342c4856b8
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. (#4852) 2022-09-19 13:14:55 -07:00
slfan1989
88545a875d
YARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-11269. (#4878) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:08:16 +09:00
Ashutosh Gupta
55d8a91b2c
YARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:05:24 +09:00
slfan1989
3ce353395b
YARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843) 2022-09-12 12:33:21 -07:00
Ashutosh Gupta
65a027b112
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-12 23:32:20 +09:00
Ashutosh Gupta
21bae31d58
YARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (#4772) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-11 03:31:28 +09:00
slfan1989
cdcb448b78
YARN-11286. Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter configurable. (#4824). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 23:00:50 +05:30
slfan1989
b2760520c3
YARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 22:57:06 +05:30
slfan1989
e76ffbf102
YARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863) 2022-09-09 10:39:00 -07:00
9uapaw
5b85af87f0 YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta. 2022-09-09 14:38:41 +02:00
slfan1989
29c4d8d8f7
YARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870) 2022-09-08 11:42:36 -07:00
slfan1989
7bf95d7949
YARN-11289. [Federation] Improve NM FederationInterceptor removeAppFromRegistry. (#4836) 2022-09-02 10:41:31 -07:00
slfan1989
1965708d49
YARN-11273. Federation StateStore: Support storage/retrieval of Reservations With SQL. (#4817) 2022-09-02 10:39:58 -07:00
slfan1989
b266f852d7
YARN-11284. [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop (#4814) 2022-09-02 10:28:38 -07:00
slfan1989
3a96de7756
YARN-6667. Handle containerId duplicate without failing the heartbeat in Federation Interceptor. (#4810) 2022-09-02 10:25:26 -07:00
slfan1989
37e213c3fc
YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation. (#4764) 2022-09-01 16:35:20 -07:00
slfan1989
33edbed54c
YARN-11272. Federation StateStore: Support storage/retrieval of Reservations With Zk. (#4781) 2022-08-31 10:15:15 -07:00
slfan1989
8a47ed6f84
YARN-11287. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-10793 (#4828) 
Co-authored-by: slfan1989 <louj1988@@>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-30 20:41:04 +09:00
Ashutosh Gupta
90dba8b614
YARN-11245. Upgrade JUnit from 4 to 5 in hadoop-yarn-csi (#4778) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-30 17:26:06 +09:00
Samrat
2c05015716
YARN-11196. NUMA support in DefaultContainerExecutor (#4742) 2022-08-30 10:39:41 +05:30
slfan1989
c60a900583
YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager. (#4792) 2022-08-27 10:17:00 -07:00
slfan1989
0075ef15c2
YARN-8482. [Router] Add cache for fast answers to getApps. (#4769) 2022-08-27 10:14:55 -07:00
slfan1989
4031b0774e
YARN-11253. Add Configuration to delegationToken RemoverScanInterval. (#4751) 2022-08-27 10:02:59 -07:00
slfan1989
f8b9dd911c
YARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router. (#4757) 2022-08-26 16:01:17 -07:00
slfan1989
75aff247ae
YARN-11240. Fix incorrect placeholder in yarn-module. (#4678). Contributed by fanshilun 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-24 20:06:36 +05:30
slfan1989
eda4bb5dcd
YARN-11250. Capture the Performance Metrics of ZookeeperFederationStateStore. (#4738) 2022-08-22 14:09:20 -07:00
Ashutosh Gupta
c294a414b9
YARN-9425. Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService (#4731). Contributed by groot and Shen Yinjie. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-22 03:40:00 +05:30
Clara Fang
c870171182
YARN-11254. hadoop-minikdc dependency duplicated in hadoop-yarn-server-nodemanager (#4755) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:09:42 +09:00
Ashutosh Gupta
b253b3be9f
YARN-11269. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timeline-pluginstorage (#4771) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 06:52:23 +09:00
slfan1989
f75c58a1ca
YARN-11252. Yarn Federation Router Supports Update / Delete Reservation in MemoryStore. (#4741) 2022-08-18 21:13:43 -07:00
slfan1989
cd72f7e042
YARN-11224. [Federation] Add getAppQueue, updateAppQueue REST APIs for Router. (#4747) 2022-08-17 13:13:07 -07:00
Ashutosh Gupta
f02ff1afe2
YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING (#4721) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-16 19:07:05 +09:00
slfan1989
eff3b8c59a
YARN-10885. Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache. (#4701) 2022-08-15 13:46:40 -07:00
slfan1989
ab88e4b65d
YARN-11223. [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router. (#4733) 2022-08-14 19:22:16 -07:00
slfan1989
d383cc4525
YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. (#4711) 2022-08-13 10:37:20 -07:00
zhengchenyu
9f6bbc90a8
YARN-11148. In federation and security mode, nm recover may fail. (#4308) 2022-08-13 10:33:16 -07:00
slfan1989
6ca2d3f848
YARN-6539. Create SecureLogin inside Router. (#4712) 2022-08-11 13:25:51 -07:00
slfan1989
133e8aabf0
YARN-11227. [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router. (#4715) 2022-08-10 14:53:46 -07:00
slfan1989
ffa9ed93a4
YARN-6572. Refactoring Router services to use common util classes for pipeline creations. (#4594) 2022-08-09 14:44:29 -07:00
Ashutosh Gupta
92abd99450
YARN-11237. Fix Bug while disabling proxy failover with Federation (#4658) 2022-08-08 13:29:27 -07:00
slfan1989
d8d3325d2f
HADOOP-18387. Fix incorrect placeholder in hadoop-common (#4679). Contributed by fanshilun. 2022-08-08 02:35:39 +05:30
Ashutosh Gupta
1cda2dcb6e
YARN-10793. Upgrade Junit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice (#4603) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-07 16:15:47 +09:00
slfan1989
52c2d99889
YARN-11228. [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router. (#4695) 2022-08-06 09:36:26 -07:00
Steve Loughran
62dbefd8f2
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.
 2022-08-05 14:06:22 +01:00
slfan1989
6f7c4c74ea
YARN-11235. Refactor Policy Code and Define getReservationHomeSubcluster (#4656) 2022-08-04 10:16:08 -07:00
slfan1989
c5ec727435
YARN-11230. [Federation] Add getContainer, signalToContainer REST APIs for Router. (#4689) 2022-08-03 11:21:48 -07:00
slfan1989
6463f86f83
YARN-11029. Refactor AMRMProxy Service code and Added Some Metrics. (#4650) 2022-08-03 09:38:00 -07:00
slfan1989
c5eba323bc
YARN-6972. Adding RM ClusterId in AppInfo. (#4673) 2022-08-03 09:35:40 -07:00
slfan1989
1f0a71a92b
YARN-8973. [Router] Add missing methods in RMWebProtocol. (#4664) 2022-08-02 14:07:09 -07:00
slfan1989
57da4bb0a1
YARN-11220. [Federation] Add getLabelsToNodes, getClusterNodeLabels, getLabelsOnNode REST APIs for Router (#4657) 2022-08-02 12:09:55 -07:00
slfan1989
2680f17eb4
YARN-11180. Refactor some code of getNewApplication, submitApplication etc. (#4618) 2022-07-29 08:23:11 -07:00
slfan1989
e994635a95
YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router. (#4614) 2022-07-28 11:53:04 -07:00
9uapaw
bf570bd4ac YARN-11063. Support auto queue creation template wildcards for arbitrary queue depths. Contributed by Bence Kosztolnik. 2022-07-28 17:32:20 +02:00