big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,652 Commits 2 Branches 2 Tags 264 MiB
d312991ead
Commit Graph

5631 Commits

Author SHA1 Message Date
Szilard Nemeth
e3e7daa4f5 YARN-9889. [UI] Add Application Tag column to RM All Applications table. Contributed by Kinga Marton 2019-10-30 16:26:56 +01:00
Abhishek Modi
ed45c13f67 YARN-2442. ResourceManager JMX UI does not give HA State. Contributed by Rohith Sharma and Cyrus Jackson. 2019-10-29 15:54:55 +05:30
Eric Badger
eef34f2d87 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 2019-10-25 17:13:04 +00:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Zhankun Tang
fd84ca5161 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:05:00 +08:00
Eric E Payne
83d148074f YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 2019-10-21 20:30:50 +00:00
Eric Yang
3f7756dc6c YARN-9922. Fixed YARN javadoc errors from YARN-9699. 
Contributed by Peter Bacsko
 2019-10-21 16:16:49 -04:00
Eric Yang
155864da00 YARN-9875. Improve fair scheduler configuration store on HDFS. 
Contributed by Prabhu Joseph
 2019-10-18 20:30:11 -04:00
Eric Yang
72b1bed998 YARN-9884. Make container-executor mount logic modular 
Contributed by Eric Badger
 2019-10-18 19:10:32 -04:00
Mate Szalay-Beko
6d92aa7c30 HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 (#1656). Contributed by Norbert Kalmár, Mate Szalay-Beko 
* HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5

- Add a static initializer for the unit tests using ZooKeeper to enable
the four-letter-words diagnostic telnet commands. (this is an interface
that become disabled by default, so to keep the ZooKeeper 3.4.x behavior
we enabled it for the tests)
- Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt
records. The new ZooKeeper seems to respond quicker during the fail-over
tests than the ZooKeeper, so we made sure to catch all the relevant records
by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().

Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>
 2019-10-18 13:26:20 -07:00
Yuming Wang
3d41f33018 HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng. 
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-18 08:25:02 -07:00
Szilard Nemeth
f0699a7406 YARN-9841. Capacity scheduler: add support for combined %user + %primary_group mapping. Contributed by Manikandan R 2019-10-18 11:19:49 +02:00
Sunil G
3990ffa083 YARN-9909. Offline format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-10-17 06:43:06 +05:30
Eric E Payne
a5034c7988 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 2019-10-16 21:10:08 +00:00
Sunil G
090f73a9aa YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 2019-10-16 18:14:31 +05:30
Szilard Nemeth
a8ef03e961 YARN-9840. Capacity scheduler: add support for Secondary Group rule mapping. Contributed by Manikandan R 2019-10-15 17:22:52 +02:00
Szilard Nemeth
5cc7873a47 YARN-9699. Migration tool that help to generate CS config based on FS config [Phase 1]. Contributed by Peter Bacsko 2019-10-14 17:19:39 +02:00
Szilard Nemeth
ec86f42e40 YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-11 14:01:19 +02:00
Eric Yang
31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN 
Contributed by Prabhu Joseph and Shane Kumpf
 2019-10-10 19:02:02 -04:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Szilard Nemeth
35f093f5b3 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 2019-10-09 13:09:09 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Sunil G
4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Sunil G
0d2d6f9267 YARN-9792. Document examples of SchedulerConf with Node Labels. Contributed by Prabhu Joseph 2019-10-02 23:43:47 +05:30
Abhishek Modi
61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
Eric Yang
d1ddba60e5 YARN-9862. Increase yarn-services-core test timeout value. 
Contributed by Prabhu Joseph
 2019-10-01 21:15:26 -04:00
Sunil G
99cd7572f1 YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 2019-10-01 20:06:21 +05:30
bibinchundatt
425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Sunil G
137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Eric Badger
18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Szilard Nemeth
c72457787d YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-09-25 10:36:22 +02:00
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Jonathan Hung
c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Jonathan Hung
3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Malcolm Taylor
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM. 
Contributed by Malcolm Taylor.

Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
 2019-09-19 16:00:05 +01:00
Sunil G
01d7924473 YARN-9814. JobHistoryServer can't delete aggregated files, if remote app root directory is created by NodeManager. Contributed by Adam Antal. 2019-09-18 09:27:41 +05:30
Weiwei Yang
f8c14326ee YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 2019-09-18 09:12:28 +08:00
Eric Yang
eefe9bc85c YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k. 
Contributed by Tarun Parimi
 2019-09-17 13:13:35 -04:00
Sunil G
c474e24c0b YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 2019-09-17 19:41:38 +05:30
Abhishek Modi
da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
HUAN-PING SU
4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
 2019-09-13 10:36:45 -07:00
Abhishek Modi
44850f6784 YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph. 2019-09-12 12:50:10 +05:30
Abhishek Modi
3b06f0bf9e YARN-9819. Make TestOpportunisticContainerAllocatorAMService more resilient. Contribued by Abhishek Modi 2019-09-12 08:09:57 +05:30
Eric Badger
56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Jonathan Hung
f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang
10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. 
Contributed by Prabhu Joseph
 2019-09-10 17:04:39 -04:00
Abhishek Modi
60af8793b4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 14:26:00 +05:30
Jonathan Hung
387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00
Jonathan Hung
43e389b980 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:07:11 -07:00
Abhishek Modi
162af6f554 YARN-9784. org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky. Contributed by Julia Kinga Marton. 2019-09-07 18:08:04 +05:30
Wangda Tan
34b82e6da0 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
 2019-09-06 19:18:13 -07:00
Tao Yang
73575701ab YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:52:39 +08:00
Jonathan Hung
b71a7f195c YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 11:00:07 -07:00
Jonathan Hung
e7d44e48f7 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:17:07 -07:00
Jonathan Hung
0ccf4b0fe1 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 2019-09-05 14:01:42 -07:00
Billie Rinaldi
2e2e5401f2 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 2019-09-05 12:49:58 -07:00
Tao Yang
172bcd8e01 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. 2019-09-05 16:20:05 +08:00
Rohith Sharma K S
3db7184082 YARN-9804. Update ATSv2 document for latest feature supports. 2019-09-05 08:42:17 +05:30
Zhankun Tang
bb26514ba9 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 2019-09-03 14:55:15 +08:00
bibinchundatt
03489124ea YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T. 2019-09-03 11:53:34 +05:30
Abhishek Modi
f4d6e82075 YARN-9400. Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId. Contributed by Prabhu Joseph. 2019-09-02 10:58:23 +05:30
Abhishek Modi
b939ae98e6 YARN-8174. Add containerId to ResourceLocalizationService fetch failure log statement. Contributed by Prabhu Joseph. 2019-09-02 10:45:59 +05:30
Abhishek Modi
bc27f7fb6d YARN-7982. Do ACLs check while retrieving entity-types per application. Contribued by Prabhu Joseph. 2019-09-02 10:15:59 +05:30
Abhishek Modi
751b5a1ac8 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 2019-09-01 22:40:40 +05:30
Abhishek Modi
fef65b4c2b YARN-9793. Remove duplicate sentence from TimelineServiceV2.md. Contributed by Julia Kinga Marton. 2019-09-01 12:43:26 +05:30
Abhishek Modi
c7ef4fb74a YARN-8678. Queue Management API - rephrase error messages. Contributed by Prabhu Joseph. 2019-09-01 10:00:04 +05:30
Abhishek Modi
d2d963f3d4 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 2019-09-01 09:54:46 +05:30
Abhishek Modi
7d998cb627 YARN-9800. TestRMDelegationTokens can fail in testRemoveExpiredMasterKeyInRMStateStore. Contributed by Adam Antal. 2019-08-31 10:03:09 +05:30
Abhishek Modi
eeccd2f6f6 YARN-9798. ApplicationMasterServiceTestBase#testRepeatedFinishApplicationMaster fails intermittently. Contributed by Tao Yang. 2019-08-31 09:57:45 +05:30
Abhishek Modi
c929b383f8 YARN-9540. TestRMAppTransitions fails intermittently. Contributed by Tao Yang. 2019-08-30 17:17:43 +05:30
Rohith Sharma K S
5b557de721 YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang. 2019-08-30 10:35:09 +05:30
Weiwei Yang
8c0759d02a YARN-9664. Improve response of scheduler/app activities for better understanding. Contributed by Tao Yang. 2019-08-29 18:14:39 +08:00
Eric E Payne
6f2226a013 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 2019-08-28 20:40:34 +00:00
Jonathan Hung
8ef46595da YARN-9438. launchTime not written to state store for running applications 2019-08-27 15:53:43 -07:00
Eric Yang
717c853873 YARN-9755. Fixed RM failing to start when FileSystemBasedConfigurationProvider is configured. 
Contributed by Prabhu Joseph
 2019-08-27 13:14:59 -04:00
Rohith Sharma K S
d70f5231a7 YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt. 2019-08-27 08:38:12 +05:30
Jonathan Hung
8660e48ca1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 2019-08-26 15:50:33 -07:00
bibinchundatt
d3ce53e507 YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt. 2019-08-26 23:21:33 +05:30
Eric Badger
6d7f01c92d YARN-9771. Add GPU in the container-executor.cfg example. Contributed by Julia Kinga Marton 2019-08-26 16:12:35 +00:00
Rohith Sharma K S
689d2e6105 YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang. 2019-08-26 21:06:15 +05:30
Szilard Nemeth
7ab88dbfa6 YARN-7291. Better input parsing for resource in allocation file. Contributed by Zoltan Siegl 2019-08-21 17:01:18 +02:00
Szilard Nemeth
e8fa192f07 YARN-9217. Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing. Contributed by Peter Bacsko 2019-08-21 16:44:22 +02:00
bibinchundatt
e684b17e6f YARN-5857. TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk. Contributed by Bilwa S T. 2019-08-21 17:14:42 +05:30
Sunil G
094d73608f YARN-9758. Upgrade JQuery to latest version for YARN UI. Contributed by Akhil PB. 2019-08-20 18:47:44 +05:30
Sunil G
0e0ddfaf24 YARN-2599. Standby RM should expose jmx endpoint. Contributed by Rohith Sharma K S. 2019-08-17 15:43:19 +05:30
HUAN-PING SU
8d754c2c39 HADOOP-16351. Change ":" to ApplicationConstants.CLASS_PATH_SEPARATOR. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-16 17:26:09 -07:00
Szilard Nemeth
9b8359bb08 YARN-9461. TestRMWebServicesDelegationTokenAuthentication.testCancelledDelegationToken fails with HTTP 400. Contributed by Peter Bacsko 2019-08-16 12:31:58 +02:00
Szilard Nemeth
4456ea67b9 YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko 2019-08-16 11:36:14 +02:00
Szilard Nemeth
2216ec54e5 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 09:13:20 +02:00
Szilard Nemeth
2a05e0ff3b YARN-9749. TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk. Contributed by Adam Antal 2019-08-16 08:52:09 +02:00
Adam Antal
22c4f38c4b YARN-9679. Regular code cleanup in TestResourcePluginManager (#1122) 2019-08-15 17:32:05 +02:00
Szilard Nemeth
1845a83cec YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph 2019-08-15 17:15:38 +02:00
HUAN-PING SU
167acd87da YARN-9683. Remove reapDockerContainerNoPid left behind by YARN-9074 (#1212) Contributed by Kevin Su. 
Reviewed-by: Eric Yang <eyang@apache.org>
Reviewed-by: Adam Antal <adam.antal@cloudera.com>
 2019-08-14 10:42:29 -07:00
Adam Antal
c89bdfacc8 YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… (#1261) 
* YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes

* Using {} placeholder, and increasing loglevel if log aggregation failed.
 2019-08-14 17:35:16 +02:00
Szilard Nemeth
3e0410449f YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. Contributed by Peter Bacsko 2019-08-14 17:13:54 +02:00
Szilard Nemeth
e5e609384f YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 16:58:22 +02:00
bibinchundatt
89a53c7eb4 YARN-9747. Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs. Contributed by Prabhu Joseph. 2019-08-14 13:46:23 +05:30
Eric Badger
2ac029b949 YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 2019-08-13 16:21:18 +00:00
Abhishek Modi
b4097b96a3 YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph. 2019-08-13 19:04:00 +05:30
Eric Yang
201dc667e9 YARN-9719. Fixed YARN service restart bug when application ID no longer exist in RM. 
Contributed by kyungwan nam
 2019-08-12 18:24:00 -04:00
Szilard Nemeth
e4b538bbda YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph 2019-08-12 15:15:43 +02:00
Szilard Nemeth
b91099efd6 YARN-9451. AggregatedLogsBlock shows wrong NM http port. Contributed by Prabhu Joseph 2019-08-12 15:05:28 +02:00
Szilard Nemeth
e0517fea33 YARN-9134. No test coverage for redefining FPGA / GPU resource types in TestResourceUtils. Contributed by Peter Bacsko 2019-08-12 14:36:07 +02:00
Abhishek Modi
13a5803ccf YARN-9464. Support pending resource metrics in RM's RESTful API. Contributed by Prabhu Joseph. 2019-08-12 14:31:24 +05:30
Abhishek Modi
8fbf8b2eb0 YARN-9722. PlacementRule logs object ID in place of queue name. Contributed by Prabhu Joseph. 2019-08-12 10:44:46 +05:30
Abhishek Modi
cf5d8957ee YARN-9657. AbstractLivelinessMonitor add serviceName to PingChecker thread. Contributed by Bilwa S T. 2019-08-11 15:41:56 +05:30
Sunil G
1c5b28659f YARN-9729. [UI2] Fix error message for logs when ATSv2 is offline. Contributed by Zoltan Siegl. 2019-08-11 11:47:07 +05:30
Sunil G
e69db45e50 Revert "YARN-9729. [UI2] Fix error message for logs without ATSv2. Contributed by Zoltan Siegl." 
This reverts commit 50cec5a704.
 2019-08-10 21:10:22 +05:30
Sunil G
50cec5a704 YARN-9729. [UI2] Fix error message for logs without ATSv2. Contributed by Zoltan Siegl. 2019-08-10 21:06:25 +05:30
Eric Yang
6ff0453ede YARN-9527. Prevent rogue Localizer Runner from downloading same file repeatly. 
Contributed by Jim Brennan
 2019-08-09 14:12:17 -04:00
Abhishek Modi
a79564fed0 YARN-9732. yarn.system-metrics-publisher.enabled=false is not honored by RM. Contributed by KWON BYUNGCHANG. 2019-08-09 22:25:30 +05:30
Sunil G
acffec7a92 YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid potential misuses. Contributed by Akhil PB. 2019-08-09 16:05:02 +05:30
Szilard Nemeth
e0c21c6da9 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak 2019-08-09 10:18:34 +02:00
Szilard Nemeth
742e30b473 YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak 2019-08-09 09:59:19 +02:00
Szilard Nemeth
72d7e570a7 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak 2019-08-09 09:49:18 +02:00
Charles Xu
9e6519a11a YARN-9711. Missing spaces in NMClientImpl (#1177) Contributed by Charles Xu. 2019-08-08 06:41:04 -07:00
Eric E Payne
3b38f2019e YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang. 2019-08-08 12:37:50 +00:00
hunshenshi
22d7d1f8bf YARN-9601.Potential NPE in ZookeeperFederationStateStore#getPoliciesConfigurations (#908) Contributed by hunshenshi. 2019-08-07 21:26:14 -07:00
Haibo Chen
f51702d539 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 2019-08-06 13:52:30 -07:00
HUAN-PING SU
7c2042a44d YARN-9678. Addendum: TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 10:21:55 -07:00
HUAN-PING SU
b8bf09ba3d YARN-9678. TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 09:05:53 -07:00
HUAN-PING SU
1c53ce0cda YARN-9410. Typo in documentation: Using FPGA On YARN (#1220) Contributed by kevin su. 2019-08-06 06:29:58 -07:00
Eric Yang
d6697da5e8 YARN-9667. Use setbuf with line buffer to reduce fflush complexity in container-executor. 
Contributed by Peter Bacsko
 2019-08-05 13:59:12 -04:00
Szilard Nemeth
54ac80176e Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph 2019-08-02 13:38:06 +02:00
Vidura Mudalige
1930a7bf60 YARN-9093. Remove commented code block from the beginning of Tes… (#444) 2019-08-02 13:16:19 +02:00
Adam Antal
cbcada804d YARN-9124. Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently (#1121) 2019-08-02 13:04:34 +02:00
Adam Antal
95fc38f2e9 YARN-9375. Use Configured in GpuDiscoverer and FpgaDiscoverer (#1131) 
Contributed by Adam Antal
 2019-08-02 11:24:09 +02:00
Eric E Payne
42683aef1a YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 2019-07-30 18:58:36 +00:00
Siyao Meng
9b8b3acb0a HADOOP-16451. Update jackson-databind to 2.9.9.1. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-24 17:25:25 -07:00
Eric Yang
9838a47d44 YARN-6046. Fixed documentation error in YarnApplicationSecurity. 
Contributed by Yousef Abu-Salah
 2019-07-18 12:39:18 -04:00
Eric Yang
c34ceb5fde YARN-9568. Fixed NPE in MiniYarnCluster during FileSystemNodeAttributeStore.recover. 
Contributed by Steve Loughran
 2019-07-18 12:30:53 -04:00
Weiwei Yang
85d9111a88 YARN-9682. Wrong log message when finalizing the upgrade. Contributed by kyungwan nam. 2019-07-17 10:28:20 +08:00
Haibo Chen
5915c902aa YARN-9646. DistributedShell tests failed to bind to a local host name. (Contributed by Ray Yang) 2019-07-16 17:36:49 -07:00
bibinchundatt
7a93be0f60 YARN-9645. Fix Invalid event FINISHED_CONTAINERS_PULLED_BY_AM at NEW on NM restart. Contributed by Bilwa S T. 2019-07-16 14:03:22 +05:30
Szilard Nemeth
5446308360 YARN-9326. Fair Scheduler configuration defaults are not documented in case of min and maxResources. Contributed by Adam Antal 2019-07-15 13:28:01 +02:00
Szilard Nemeth
18ee1092b4 YARN-9127. Create more tests to verify GpuDeviceInformationParser. Contributed by Peter Bacsko 2019-07-15 11:59:11 +02:00
Szilard Nemeth
91ce09e706 YARN-9360. Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource. Contributed by Peter Bacsko 2019-07-15 10:47:20 +02:00
Szilard Nemeth
557056e18e YARN-9626. UI2 - Fair scheduler queue apps page issues. Contributed by Zoltan Siegl 2019-07-12 17:40:14 +02:00
Szilard Nemeth
61b0c2bb7c YARN-9337. GPU auto-discovery script runs even when the resource is given by hand. Contributed by Adam Antal 2019-07-12 17:28:14 +02:00
Szilard Nemeth
8b3c6791b1 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-07-12 17:20:42 +02:00
Szilard Nemeth
c416284bb7 YARN-9235. If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown. Contributed by Antal Balint Steinbach, Adam Antal 2019-07-12 16:51:58 +02:00
Haibo Chen
9b54dd7186 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonathan Hung) 2019-07-11 13:57:08 -07:00
Szilard Nemeth
9cec023186 YARN-9625. UI2 - No link to a queue on the Queues page for Fair Scheduler. Contributed by Zoltan Siegl 2019-07-11 16:11:21 +02:00
Akira Ajisaka
ccaa99c923
HADOOP-16381. The JSON License is included in binary tarball via azure-documentdb:1.16.2. Contributed by Sushil Ks. 2019-07-11 13:49:42 +09:00
Eric Yang
555dabf4f8 YARN-9660. Update support documentation for Docker on YARN. 
Contributed by Peter Bacsko
 2019-07-10 17:15:33 -04:00
Szilard Nemeth
a2a8be18cb YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by Adam Antal. 2019-07-03 13:45:00 +02:00
Weiwei Yang
15d82fcb75 YARN-9658. Fix UT failures in TestLeafQueue. Contributed by Tao Yang. 2019-07-03 12:08:45 +08:00
Sunil G
e966edd025 YARN-9644. First RMContext object is always leaked during switch over. Contributed by Bibin A Chundatt. 2019-07-02 12:18:16 +05:30
Sunil G
0c8813f135 YARN-9327. Improve synchronisation in ProtoUtils#convertToProtoFormat block. Contributed by Bibin A Chundatt. 2019-07-02 11:44:11 +05:30
Weiwei Yang
570eee30e5 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi. 2019-07-02 09:55:25 +08:00
hunshenshi
b1dafc3506 YARN-9661:Fix typo in LocalityMulticastAMRMProxyPolicy.java and AbstractConfigurableFederationPolicy.java (#1042) 2019-07-01 10:46:33 -07:00
Eric Yang
29465bf169 YARN-9560. Restructure DockerLinuxContainerRuntime to extend OCIContainerRuntime. 
Contributed by Eric Badger, Jim Brennan, Craig Condit
 2019-06-28 17:18:53 -04:00
Eric Yang
f02b0e1994 YARN-9581. Add support for get multiple RM webapp URLs. 
Contributed by Prabhu Joseph
 2019-06-28 14:54:57 -04:00
Weiwei Yang
f09c31a97e Revert "YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023" 
This reverts commit 5e7caf1287.
 2019-06-29 00:29:17 +08:00
Weiwei Yang
5e7caf1287 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023 2019-06-29 00:08:40 +08:00
Weiwei Yang
cbae241320 YARN-9623. Auto adjust max queue length of app activities to make sure activities on all nodes can be covered. Contributed by Tao Yang. 2019-06-28 23:24:53 +08:00
bibinchundatt
be80334cdf YARN-9639. DecommissioningNodesWatcher cause memory leak. Contributed by Bilwa S T. 2019-06-27 09:59:44 +05:30
Giovanni Matteo Fumarola
1ac967a6b7 YARN-6055. ContainersMonitorImpl need be adjusted when NM resource changed. Contributed by Inigo Goiri. 2019-06-26 14:01:31 -07:00
Zhankun Tang
062eb605ac YARN-9477. Implement VE discovery using libudev. Contributed by Peter Bacsko. 2019-06-26 23:53:14 +08:00
Eric Yang
b220ec6f61 YARN-9374. Improve Timeline service resilience when HBase is unavailable. 
Contributed by Prabhu Joseph and Szilard Nemeth
 2019-06-24 12:19:14 -04:00
Weiwei Yang
83dcb9d87e YARN-9209. When nodePartition is not set in Placement Constraints, containers are allocated only in default partition. Contributed by Tarun Parimi. 2019-06-21 17:41:05 +08:00
Eric Yang
5bfdf62614 YARN-9631. Added ability to select JavaScript test or skip JavaScript tests for YARN application catalog. 
Contributed by Eric Yang

(cherry picked from commit 6002b0c5c6994965d3f7231330248c093869dba2)
 2019-06-19 18:48:16 -04:00
Eric Yang
b14f0569bb YARN-9574. Update hadoop-yarn-applications-mawo artifactId to match directory name. 
Contributed by Wanqiang Ji

(cherry picked from commit 5f758a69ede8fafd214857a74f5d3b46198094c4)
 2019-06-18 13:51:12 -04:00
Wanqiang Ji
eb6be4643f YARN-9630. [UI2] Add a link in docs's top page 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-06-18 14:54:21 +09:00
Zhankun Tang
67414a1a80 YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 2019-06-18 12:23:52 +08:00
Zhankun Tang
304a47e22c YARN-9608. DecommissioningNodesWatcher should get lists of running applications on node from RMNode. Contributed by Abhishek Modi. 2019-06-17 17:09:56 +08:00
Eric Yang
cda9f33745 YARN-8499 ATSv2 Generalize TimelineStorageMonitor. 
Contributed by Prabhu Joseph
 2019-06-14 18:59:14 -04:00
Eric Yang
3ba090f436 HADOOP-16366. Fixed ProxyUserAuthenticationFilterInitializer for timeline server. 
Contributed by Prabhu Joseph
 2019-06-14 12:54:16 -04:00
Giovanni Matteo Fumarola
bcfd228336 YARN-9599. TestContainerSchedulerQueuing#testQueueShedding fails intermittently. Contributed by Abhishek Modi. 2019-06-13 11:08:35 -07:00
Weiwei Yang
970b0b0c02 YARN-9578. Add limit/actions/summarize options for app activities REST API. Contributed by Tao Yang. 2019-06-13 10:44:47 +08:00
Eric Yang
205dd2d8e1 HADOOP-16367. Fixed MiniYarnCluster AuthenticationFilter initialization. 
Contributed by Prabhu Joseph
 2019-06-12 18:03:33 -04:00
bibinchundatt
2263ead365 YARN-9557. Application fails in diskchecker when ReadWriteDiskValidator is configured. Contributed by Bilwa S T. 2019-06-11 23:20:28 +05:30
bibinchundatt
60c95e9b6a YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T. 2019-06-11 23:11:49 +05:30
bibinchundatt
6d80b9bc3f YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 2019-06-11 22:49:21 +05:30
bibinchundatt
f7df55f4a8 YARN-9602. Use logger format in Container Executor. Contributed by Abhishek Modi. 2019-06-11 22:29:00 +05:30
Suma Shivaprasad
9191e08f0a YARN-9569. Auto-created leaf queues do not honor cluster-wide min/max memory/vcores. Contributed by Craig Condit. 2019-06-10 14:33:24 -07:00
Adam Antal
e94e643584 YARN-9471. Cleanup in TestLogAggregationIndexFileController. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 12:18:15 -07:00
Eric Yang
cb9bc6e64c YARN-9581. Fixed yarn logs cli to access RM2. 
Contributed by Prabhu Joseph
 2019-06-06 16:41:58 -04:00
Weiwei Yang
0976392502 YARN-9590. Correct incompatible, incomplete and redundant activities. Contributed by Tao Yang. 2019-06-06 21:59:01 +08:00
Sunil G
649666e118 YARN-9573. DistributedShell cannot specify LogAggregationContext. Contributed by Adam Antal. 2019-06-06 14:51:55 +05:30
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Weiwei Yang
433e97cd34 YARN-9600. Support self-adaption width for columns of containers table on app attempt page. Contributed by Tao Yang. 2019-06-05 13:55:30 +08:00
Eric Yang
d45669cd3c YARN-7537. Add ability to load hbase config from distributed file system. 
Contributed by Prabhu Joseph
 2019-06-04 19:26:06 -04:00
Zhankun Tang
606061aa14 YARN-9595. FPGA plugin: NullPointerException in FpgaNodeResourceUpdateHandler.updateConfiguredResource(). Contributed by Peter Bacsko. 2019-06-04 09:56:59 +08:00
Weiwei Yang
bd2590d71b YARN-9580. Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers. Contributed by Tao Yang. 2019-06-03 22:59:02 +08:00
Sunil G
59719dc560 YARN-8906. [UI2] NM hostnames not displayed correctly in Node Heatmap Chart. Contributed by Akhil PB. 2019-06-03 15:53:23 +05:30
Sunil G
7f46dda513 YARN-8947. [UI2] Active User info missing from UI2. Contributed by Akhil PB. 2019-06-03 12:24:35 +05:30
Weiwei Yang
4530f4500d YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T. 2019-06-03 14:09:37 +08:00
Giovanni Matteo Fumarola
2210897609 YARN-9592. Use Logger format in ContainersMonitorImpl. Contributed by Inigo Goiri. 2019-05-31 17:35:49 -07:00
Eric Yang
4cb559ea7b YARN-9027. Fixed LevelDBCacheTimelineStore initialization. 
Contributed by Prabhu Joseph
 2019-05-31 14:31:44 -04:00
Sunil G
52128e352a YARN-9543. [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2. Contributed by Zoltan Siegl and Akhil P B. 2019-05-31 12:29:44 +05:30
Sunil G
e49162f4b3 YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 2019-05-31 10:28:09 +05:30
Sunil G
7861a5eb1a YARN-9033. ResourceHandlerChain#bootstrap is invoked twice during NM start if LinuxContainerExecutor enabled. Contributed by Zhankun Tang. 2019-05-31 10:22:26 +05:30
Giovanni Matteo Fumarola
f1552f6edb YARN-9553. Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines. Contributed by Prabhu Joseph. 2019-05-30 11:42:27 -07:00