big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,116 Commits 2 Branches 2 Tags 264 MiB
37e213c3fc
Commit Graph

6507 Commits

Author SHA1 Message Date
Ashutosh Gupta
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping (#4515) 
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-02 21:58:56 +05:30
Tamas Domok
3cad632709 YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok. 
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
 2022-07-01 10:50:48 +02:00
9uapaw
2d133a54ac YARN-11204. Various MapReduce tests fail with NPE in AggregatedLogDeletionService.stopRMClient. Contributed by Szilard Nemeth. 2022-06-29 15:12:04 +02:00
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Samrat
7eefdf8642
YARN-11195. Adding document to enable numa (#4501) 
Contributed by Samrat Deb.
 2022-06-28 17:46:43 +05:30
Ashutosh Gupta
a177232ebc
YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down (#4492) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-28 09:32:07 +05:30
Ashutosh Gupta
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation (#4487) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-23 21:59:27 +05:30
Ashutosh Gupta
4abb2ba58c
YARN-10320.Replace FSDataInputStream#read with readFully in Log Aggregation (#4486) 
* YARN-10320.Replace FSDataInputStream#read with readFully in Log Aggregation

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-23 21:58:32 +05:30
slfan1989
0af4bb3b42
YARN-11192. TestRouterWebServicesREST failing after YARN-9827. (#4484). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-23 13:21:36 +05:30
9uapaw
e6ecc4f3e4 YARN-11188. Only files belong to the first file controller are removed even if multiple log aggregation file controllers are configured. Contributed by Szilard Nemeth. 2022-06-22 14:40:20 +02:00
Ashutosh Gupta
cbdabe9ec8
YARN-9971.YARN Native Service HttpProbe logs THIS_HOST in error messages (#4436) 
* YARN-9971.YARN Native Service HttpProbe logs THIS_HOST in error messages

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-22 09:38:19 +05:30
Szilard Nemeth
3a66348fda YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori 2022-06-20 15:03:58 +02:00
9uapaw
5d08ffa769 YARN-11182. Refactor TestAggregatedLogDeletionService: 2nd phase. Contributed by Szilard Nemeth. 2022-06-20 14:12:51 +02:00
Ashutosh Gupta
4f425b641c
YARN-9827.Fix Http Response code in GenericExceptionHandler (#4393) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed by Akira Ajisaka.
 2022-06-20 10:44:47 +05:30
slfan1989
62e4476102
YARN-10122. Support signalToContainer API for Federation. (#4421) 2022-06-17 16:38:36 -07:00
zhengchenyu
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. (#4408) 
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.

Contributed by zhengchenyu
 2022-06-17 19:49:36 +01:00
Benjamin Teke
020201cb65 Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu. 2022-06-17 19:28:32 +02:00
Gautham B A
dc5460d525
YARN-11078. Set env vars in a cross platform compatible way (#4432) 
* Maven runs the ember build script.
  The environment variable TMPDIR was
  set as per bash syntax.
* This failed on Windows since the
  Windows command prompt doesn't
  support bash syntax.
* We're now detecting the OS and
  setting a Maven property
  "emberBuildScript" in a cross
  platform compatible way.
 2022-06-15 15:29:55 +05:30
9uapaw
75bc6cfced YARN-11176. Refactor TestAggregatedLogDeletionService. Contributed by Szilard Nemeth. 2022-06-14 16:14:44 +02:00
9uapaw
c9a174a260 YARN-11175. Refactor LogAggregationFileControllerFactory 2022-06-13 13:58:13 +02:00
Steve Loughran
985161f6ec
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi (#4417) 
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.

Contributed by Steve Loughran
 2022-06-09 13:10:46 +01:00
slfan1989
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation (#4396) 2022-06-08 17:34:43 -07:00
slfan1989
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation (#4341) 2022-06-02 12:54:29 -07:00
Ashutosh Gupta
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project (#4381). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 23:17:11 +05:30
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
Ashutosh Gupta
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-26 15:51:55 +09:00
slfan1989
678768035d
YARN-11137. Improve log message in FederationClientInterceptor (#4336) 2022-05-24 17:21:04 -07:00
Owen O'Malley
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 2022-05-24 14:07:19 +09:00
slfan1989
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation (#4317) 2022-05-20 13:19:42 -07:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
9uapaw
0e6a6d1880 YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation 2022-05-18 18:22:42 +02:00
9uapaw
54cd0174c0 YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe 2022-05-18 18:16:34 +02:00
Steve Loughran
6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:54 +01:00
Zilong Zhu
79cc0caf9f
YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280) 
* Update QueueConfigurationsPBImpl.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java
 2022-05-16 22:10:09 +05:30
Ashutosh Gupta
931abbd927
YARN-11092. Upgrade jquery ui to 1.13.1 (#4260) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-16 23:11:17 +09:00
slfan1989
6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274) 2022-05-15 09:16:06 -07:00
Ashutosh Gupta
6985f9aabe
YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:41:06 +09:00
Jian Chen
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110) 
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:11:42 +09:00
Szilard Nemeth
eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Szilard Nemeth
5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
Szilard Nemeth
9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth
eeedc0c7e4 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-11 14:39:42 +02:00
Szilard Nemeth
b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989
09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
jteagles
d4a91bd0c0
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:14:26 +09:00
Ashutosh Gupta
a74acc755e
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:08:30 +09:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
Ashutosh Gupta
006767eb94
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md (#4221) 2022-04-26 09:51:11 +08:00
Steve Loughran
6999acf520
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
 2022-04-24 17:33:05 +01:00
Junfan Zhang
b35b744670
YARN-11111. Recovery failure when node-label configure-type transit from delegated-centralized to centralized (#4200) 2022-04-21 22:37:28 -04:00
章锡平
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. (#4175) 2022-04-15 17:50:49 +09:00
章锡平
3b46aae977 YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1 2022-04-13 15:57:18 +02:00
Junfan Zhang
e044a46f97 YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang 2022-03-31 12:39:56 +02:00
Junfan Zhang
6e00a799e7 YARN-11106. Fix the test failure due to missing conf of yarn.resourcemanager.node-labels.am.default-node-label-expression. Contributed by Junfan Zhang 2022-03-30 19:43:23 +02:00
PJ Fanning
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-26 20:31:16 +09:00
Tamas Domok
da09d68056 YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok 2022-03-25 21:53:18 +01:00
9uapaw
921267ca31 YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang. 2022-03-23 08:57:00 +01:00
9uapaw
2beb7296fb YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang. 2022-03-22 13:14:43 +01:00
9uapaw
c3124a3f6e YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke. 2022-03-21 14:55:49 +01:00
Szilard Nemeth
e2701e227f YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang. 2022-03-21 14:37:10 +01:00
Szilard Nemeth
4537b34e1c YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang. 2022-03-21 14:18:27 +01:00
Szilard Nemeth
ed65aa2324 YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori 2022-03-10 22:22:58 +01:00
Szilard Nemeth
481da19494 YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke 2022-03-10 22:15:35 +01:00
9uapaw
383b73417d YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-03-10 13:11:19 +01:00
9uapaw
9539ff108a YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth 2022-03-10 08:40:43 +01:00
Szilard Nemeth
db8ae4b654 YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori 2022-03-09 19:36:09 +01:00
Szilard Nemeth
2ece95064b YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori 2022-03-09 19:29:53 +01:00
Viraj Jasani
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing (#4027) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-08 21:32:51 +09:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
9uapaw
851a4dc2d9 YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth 2022-03-07 09:01:18 +01:00
Szilard Nemeth
379baa5eb6 YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-03-03 16:44:12 +01:00
Szilard Nemeth
1c27c69f44 Revert "YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori" 
This reverts commit de526e166c.

With this commit, the patch of
https://issues.apache.org/jira/browse/YARN-10850 has been committed.
 2022-03-03 16:38:56 +01:00
Tamas Domok
22fe79cee3
YARN-11076. Upgrade jQuery version in Yarn UI2. (#4046) 
Change-Id: I3cb1677741df5a1978e83029443d4a2d5d7e3d7f
 2022-03-03 23:54:03 +09:00
Szilard Nemeth
356d337d1e YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok 2022-03-02 16:19:17 +01:00
9uapaw
9e475aede6 YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke 2022-03-02 11:16:24 +01:00
Szilard Nemeth
365375412f YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori 2022-02-22 12:26:35 +01:00
Szilard Nemeth
0463498adc YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori 2022-02-22 11:55:34 +01:00
Szilard Nemeth
de526e166c YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-02-17 21:05:01 +01:00
Szilard Nemeth
7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth
51001fc7bf YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok 2022-02-17 19:38:18 +01:00
Szilard Nemeth
48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
Akira Ajisaka
7fd90cdcbe
YARN-10788. TestCsiClient fails (#3989) 
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 01:13:13 +09:00
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741) 
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-02-01 12:18:54 +05:30
Akira Ajisaka
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2022-01-28 15:51:21 +09:00
Akira Ajisaka
02f6bad1ff Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936)" 
This reverts commit 1c01944f35.
 2022-01-28 00:36:25 +09:00
Wei-Chiu Chuang
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 00:04:01 +09:00
minni31
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse (#3766) 2022-01-25 09:25:18 -08:00
Andrew Chung
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container (#3779) 2022-01-24 08:03:36 -08:00
dependabot[bot]
dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui (#3890) 
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-20 21:44:10 +09:00
Gera Shegalov
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-17 16:20:48 +09:00
luoyuan3471
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Andras Gyori
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
minni31
3bf014d871
YARN-8859. Add audit logs for router service (#3607) 
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-01-03 10:38:59 +05:30