big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,812 Commits 2 Branches 2 Tags 264 MiB
df0381f236
Commit Graph

6519 Commits

Author SHA1 Message Date
Xianming Lei
ee94f6cdcb
YARN-11277. Trigger log-dir deletion by size for NonAggregatingLogHandler. (#4797) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-06-05 11:08:06 +08:00
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
slfan1989
86c250a54a
YARN-7720. Race condition between second app attempt and UAM timeout when first attempt node is down. (#5672) 2023-05-29 10:37:08 -07:00
Xianming Lei
97afb33c73
YARN-11276. Add LRU cache for RMWebServices.getApps. (#4793) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-05-26 20:46:00 +08:00
slfan1989
b977065cc4
YARN-11478. [Federation] SQLFederationStateStore Support Store ApplicationSubmitData. (#5663) 2023-05-24 11:43:20 -07:00
Tamas Domok
aeb3f6f1a8
YARN-11490. Reverting YARN-11211 and eliminating the use of DefaultMetricsSystem during configuration validation (#5644) 2023-05-23 10:36:37 +02:00
Xianming Lei
0110e24ed8
YARN-11496. Improve TimelineService log format. (#5677). Contributed by Xianming Lei. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 14:57:45 +05:30
NishthaShah
5272ed8670
HADOOP-17518. Update the regex to A-Z (#5669). Contributed by Nishtha Shah. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-20 06:21:13 +05:30
slfan1989
bba663038d
YARN-8898. Fix FederationInterceptor#allocate to set application priority in allocateResponse. (#5645) 2023-05-18 11:57:38 -07:00
Peter Szucs
ff8eac517a
YARN-11463. Node Labels root directory creation doesn't have a retry logic - 2nd addendum (#5670) 2023-05-18 14:48:43 +02:00
slfan1989
5ddaf2e133
YARN-11493. [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router. (#5651) 2023-05-17 11:09:10 -07:00
Chun Chen
11af08d67a
YARN-11489. Fix memory leak of DelegationTokenRenewer futures in DelegationTokenRenewerPoolTracker. (#5629). Contributed by Chun Chen. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:38:04 +08:00
slfan1989
e0938b4c2a
YARN-11495. Fix typos in hadoop-yarn-server-web-proxy. (#5652). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 11:41:38 +05:30
slfan1989
2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). (#5625) 2023-05-12 18:12:05 -07:00
susheel-gupta
0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade (#5649) 2023-05-12 14:20:26 +02:00
Steve Loughran
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
slfan1989
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00
cxzl25
be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist (#5565) 2023-05-10 15:16:33 +08:00
slfan1989
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. (#5363) 2023-05-09 16:17:23 -07:00
slfan1989
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. (#5403) 2023-05-08 10:13:09 -07:00
slfan1989
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616) 2023-05-05 13:43:40 -07:00
slfan1989
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. (#5609) 2023-05-05 13:41:38 -07:00
Szilard Nemeth
c7699d3dcd YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta 2023-05-04 22:16:18 -04:00
Peter Szucs
bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum (#5614) 2023-05-04 12:27:25 +02:00
slfan1989
476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. (#5587) 2023-05-03 09:21:56 -07:00
slfan1989
c1d10f3872
YARN-9049. Add application submit data to state store. (#5606) 2023-05-03 09:19:54 -07:00
slfan1989
668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. (#5596) 2023-05-02 13:46:06 -07:00
slfan1989
87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. (#5598) 2023-05-01 16:01:50 -07:00
Hexiaoqiao
70c0aa342e
YARN-11482. Fix bug of DRF comparision DominantResourceFairnessComparator2 in fair scheduler. (#5607). Contributed by Xiaoqiao He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2023-04-29 11:18:42 +08:00
slfan1989
5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. (#5570) 
Co-authored-by: slfan1989 <louj1988@@>
 2023-04-28 14:11:13 -07:00
Riya Khandelwal
60a7e8acaa
YARN-11459 Changed label called "max resource" on UIv1 and UIv2 (#5527) 2023-04-27 15:25:25 +02:00
yl09099
245fde17d7
YARN-11474.The yarn queue list is displayed on the CLI (#5577) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-27 18:05:48 +08:00
slfan1989
55eebcf277
YARN-11378. [Federation] Support checkForDecommissioningNodes、refreshClusterMaxPriority API's for Federation. (#5551) 2023-04-25 14:12:38 -07:00
slfan1989
a716459cdf
YARN-11437. [Federation] SQLFederationStateStore Support Version. (#5589) 2023-04-25 14:09:45 -07:00
Ashutosh Gupta
964c1902c8
YARN-11463. Node Labels root directory creation doesn't have a retry logic (#5562) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2023-04-21 14:03:22 +02:00
rdingankar
5119d0c72f
HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot (#5556) 2023-04-18 10:47:37 -07:00
slfan1989
a258f1f235
YARN-11326. [Federation] Add RM FederationStateStoreService Metrics. (#4963) 2023-04-18 09:13:08 -07:00
slfan1989
635521db4c
YARN-11438. [Federation] ZookeeperFederationStateStore Support Version. (#5537) 2023-04-18 09:05:52 -07:00
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
yl09099
2c4d6bf33d
YARN-11465. Improved YarnClient Log Format (#5550) 
Co-authored-by: yl09099 <shaq376260428@163.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-17 09:27:52 +08:00
slfan1989
0bcdea7912
YARN-11239. Optimize FederationClientInterceptor audit log. (#5127) 2023-04-14 13:09:18 -07:00
slfan1989
06f9bdffa6
YARN-10846. Add dispatcher metrics to NM. (#4687) 2023-04-12 09:53:20 -07:00
slfan1989
dd6d0ac510
YARN-11462. Fix Typo of hadoop-yarn-common. (#5539) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-12 11:08:23 +08:00
slfan1989
bffa49a64f
YARN-11377. [Federation] Support addToClusterNodeLabels、removeFromClusterNodeLabels、replaceLabelsOnNode API's for Federation. (#5525) 2023-04-11 09:47:58 -07:00
slfan1989
69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
slfan1989
eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00
slfan1989
5bc8f25327
YARN-11446. [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. (#5476) 2023-03-28 09:33:19 -07:00
slfan1989
aa602381c5
YARN-11426. Improve YARN NodeLabel Memory Display. (#5335) 
YARN-11426. Improve YARN NodeLabel Memory Display.
Co-authored-by: slfan1989 <louj1988@@>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-03-28 22:48:46 +08:00
slfan1989
926993cb73
YARN-11376. [Federation] Support updateNodeResource、refreshNodesResources API's for Federation. (#5496) 2023-03-27 09:27:21 -07:00
Tamas Domok
69748aae32
YARN-11461. fix NPE in determineMissingParents (auto queue creation / CS). (#5506) 
Change-Id: Iaaaf43a545588eaff8a0a20f6f3c27258a45f390
 2023-03-24 09:38:53 +01:00
slfan1989
fa723ae839
YARN-11445. [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. (#5472) 2023-03-17 11:59:45 -07:00
Viraj Jasani
cf4a678ce9
HADOOP-18649. CLA and CRLA appenders to be replaced with RFA (#5448) 2023-03-16 00:46:17 +08:00
slfan1989
b406060c6b
YARN-8972. [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size. (#5382) 2023-03-08 13:29:30 -08:00
Varun Saxena
2a0dc2ab2f YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary
 2023-03-05 21:25:16 +05:30
slfan1989
bcc51ce2c5
YARN-11375. [Federation] Support refreshAdminAcls、refreshServiceAcls API's for Federation. (#5312) 2023-02-28 14:44:00 -08:00
slfan1989
8798b94ee1
YARN-11221. [Federation] Add replaceLabelsOnNodes, replaceLabelsOnNode REST APIs for Router. (#5302) 2023-02-27 09:34:39 -08:00
slfan1989
25ebd0b8b1
YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router. (#5328) 2023-02-24 10:52:57 -08:00
slfan1989
27a54955f9
YARN-5604. [Federation] Add versioning for FederationStateStore. (#5394) 2023-02-24 10:51:19 -08:00
slfan1989
2e997d818d
YARN-11370. [Federation] Refactor MemoryFederationStateStore code. (#5126) 2023-02-22 12:37:35 -08:00
susheel-gupta
49b8ac19f2
YARN-11408. Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method (#5278) 
* added a check of autoQueueCreationV2Disabled

Change-Id: If1e36c5969d270c1b81a4bbd2e883fa819c81f20

* added check of AutoCreateChildQueueDisabled

Change-Id: Ia011b4393ccd8d4d419a2e46b06a5237d050851c

* removed auto-create-child-queue-enabled check and emit

Change-Id: I7a154124519ecbd81379b46a238707c16db1e82a
 2023-02-22 09:46:42 +01:00
slfan1989
4e6e2f318c
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-02-21 14:39:32 +00:00
slfan1989
7e486038ea
YARN-11439. Fix Typo of hadoop-yarn-ui README.md. (#5405) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:14:01 +08:00
Steve Loughran
10e7ca481c
YARN-11441. Revert YARN-10495. 
This reverts commit 7d3c8ef606.
 2023-02-17 15:05:06 +00:00
slfan1989
a5f48eacca
YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) 2023-02-15 14:40:34 -08:00
slfan1989
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Viraj Jasani
4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989
af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
slfan1989
a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
jokercurry
dad73b76c0
YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc (#5309) 
Co-authored-by: smallzhongfeng <982458633@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-04 10:29:19 +08:00
Masatake Iwasaki
a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
slfan1989
468135a4d9
YARN-11218. [Federation] Add getActivities, getBulkActivities REST APIs for Router. (#5284) 2023-01-26 11:14:05 -08:00
Szilard Nemeth
cf1b3711cb YARN-10965. Centralize queue resource calculation based on CapacityVectors. Contributed by Andras Gyori 2023-01-26 19:45:54 +01:00
Szilard Nemeth
815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth
29f2230cb6 YARN-5607. Document TestContainerResourceUsage#waitForContainerCompletion. Contributed by Susheel Gupta 2023-01-25 15:13:24 +01:00
slfan1989
3f767a61b1
YARN-8900. [Follow Up] Fix FederationInterceptorREST#invokeConcurrent Inaccurate Order of Subclusters. (#5260) 2023-01-19 17:13:55 -08:00
slfan1989
442a5fb285
YARN-11320. [Federation] Add getSchedulerInfo REST APIs for Router. (#5217) 2023-01-17 09:36:19 -08:00
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
slfan1989
168fa07801
YARN-11409. Fix Typo of ResourceManager#webapp moudle. (#5285) 2023-01-13 11:53:13 +08:00
slfan1989
4520448327
YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. (#5193) 2023-01-12 15:04:09 -08:00
slfan1989
3d21cff263
YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. (#5289) 
* YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

* YARN-11413. Fix CheckStyle.

* YARN-11413. Fix CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2023-01-12 14:29:05 +01:00
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
Riya Khandelwal
dd49077aed
YARN-6412 aux-services classpath not documented (#5242) 2023-01-09 15:30:24 +01:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-27 15:58:53 -08:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:15:36 +01:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:20:53 +01:00
slfan1989
6172c3192d
YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. (#5056) 2022-12-14 14:37:56 -08:00
slfan1989
63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. (#5131) 2022-12-14 09:09:38 -08:00
slfan1989
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. (#5182) 2022-12-10 15:03:49 -08:00
K0K0V0K
ee7d1787cd
YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-08 09:52:19 -08:00
slfan1989
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. (#5146) 2022-12-06 08:17:05 -08:00
Gautham B A
dadd3d9138
YARN-11386. Fix issue with classpath resolution (#5183) 
* This PR ensures that all the special notations such as
  <CPS> are resolved before getting added to classpath.
 2022-12-06 16:32:26 +05:30
slfan1989
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. (#5179) 2022-12-02 10:56:17 -08:00
slfan1989
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. (#5104) 2022-12-01 13:20:21 -08:00
Szilard Nemeth
5440c75c4a YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs 2022-12-01 15:11:58 +01:00