big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
18,259 Commits 2 Branches 2 Tags 264 MiB
2d06d885c8
Commit Graph

954 Commits

Author SHA1 Message Date
Miklos Szegedi
a68e445dc6 YARN-7717. Add configuration consistency for module.enabled and docker.privileged-containers.enabled. Contributed by Eric Badger. 2018-01-17 14:11:14 -08:00
Miklos Szegedi
41049ba5d1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-16 15:40:43 -08:00
Miklos Szegedi
2dcfc1876e YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 2018-01-12 17:00:01 -08:00
Wangda Tan
edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Eric Yang
3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Miklos Szegedi
bc285da107 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 2018-01-11 09:40:09 -08:00
Jason Lowe
d795661868 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:15:52 -06:00
Haibo Chen
2f6c038be6 YARN-7602. NM should reference the singleton JvmMetrics instance. 2018-01-03 09:41:26 -08:00
Arun Suresh
c0c7cce81d YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 2018-01-02 22:03:00 -08:00
Miklos Szegedi
7f515f57ed YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 2018-01-02 18:03:04 -08:00
Miklos Szegedi
b82049b4f0 YARN-7580. ContainersMonitorImpl logged message lacks detail when exceeding memory limits. Contributed by Wilfred Spiegelenburg. 2017-12-29 12:49:37 -08:00
Arun Suresh
a55884c68e YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 2017-12-28 22:20:42 -08:00
Jason Lowe
811fabdebe YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 2017-12-18 15:20:06 -06:00
Akira Ajisaka
001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Wangda Tan
44825f0960 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
 2017-12-15 15:40:56 -08:00
Wangda Tan
631b5c2db7 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
 2017-12-15 15:38:36 -08:00
Chen Liang
46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Weiwei Yang
7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe
2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jason Lowe
06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Subru Krishnan
670e8d4ec7 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 2017-12-08 15:39:18 -08:00
Wangda Tan
04b84da245 YARN-7443. Add native FPGA module support to do isolation with cgroups. (Zhankun Tang via wangda) 
Change-Id: Ic4b7f9f3e032986b8f955139c9fe4d3a6c818a53
 2017-12-08 15:18:22 -08:00
Weiwei Yang
e411dd6666 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:29:40 +08:00
Weiwei Yang
05c347fe51 YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 2017-12-06 12:21:52 +08:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Sunil G
81f6e46b2f YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 2017-12-04 11:27:23 +05:30
Sunil G
30f2646b15 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2017-12-04 10:45:07 +05:30
Jason Lowe
60f95fb719 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 2017-12-01 15:47:01 -06:00
Wangda Tan
7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Sunil G
556aea3f36 YARN-7487. Ensure volume to include GPU base libraries after created by plugin. Contributed by Wangda Tan. 2017-12-01 13:36:28 +05:30
Wangda Tan
c9a54aab6b YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2017-11-29 17:46:16 -08:00
Yufei Gu
d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Sunil G
aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Eric Yang
d42a336cfa YARN-5534. Allow user provided Docker volume mount list. (Contributed by Shane Kumpf) 2017-11-22 13:05:34 -05:00
Subru Krishnan
d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Junping Du
e14f03dfbf YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 15:27:37 -08:00
Arun Suresh
cb35a59589 YARN-7343. Add a junit test for ContainerScheduler recovery. (Sampada Dehankar via asuresh) 2017-11-08 08:14:02 -08:00
Jian He
a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Billie Rinaldi
ce74e64363 YARN-7210. Some NPE fixes in Registry DNS. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Jian He
8d335e59cf YARN-6804. [yarn-native-services changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-11-06 13:30:10 -08:00
Billie Rinaldi
ce05c6e981 YARN-6545. Followup fix for YARN-6405. Contributed by Jian He 2017-11-06 13:30:07 -08:00
bibinchundatt
dcd99c4b9a Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 2017-11-06 22:39:10 +05:30
Jason Lowe
d00b6f7c1f YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 2017-11-02 09:37:17 -05:00
Jian He
0cc98ae0ec YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2017-11-01 17:00:32 -07:00
Eric Yang
7a49ddfdde YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:39:56 -04:00
Sunil G
9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Arun Suresh
9c5c68745e YARN-7299. Fix TestDistributedScheduler. (asuresh) 2017-10-27 23:08:18 -07:00
Jason Lowe
665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Haibo Chen
480187aebb YARN-7372. TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic is flaky. 2017-10-20 14:24:17 -07:00
Yufei Gu
0799fde35e YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 2017-10-20 10:00:13 -07:00
Eric Yang
b61144a93d YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-20 12:02:06 -04:00
Wangda Tan
c1b08ba720 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2017-10-19 14:45:44 -07:00
Nathan Roberts
4540ffd15f YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe. 2017-10-16 17:00:38 -05:00
Arun Suresh
a50be1b8f4 YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh) 2017-10-16 13:12:15 -07:00
Sunil G
fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Naganarasimha
d0046bb5c6 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 2017-10-06 00:07:53 +05:30
Sunil G
592bf2d550 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton. 2017-10-05 10:52:00 +05:30
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Sidharta S
7eb8499996 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-02 19:04:49 -07:00
Wangda Tan
febeead5f9 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
 2017-09-29 16:39:46 -07:00
Wangda Tan
d3b1c63195 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
 2017-09-28 16:41:09 -07:00
Subru Krishnan
ca669f9f8b YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 2017-09-28 13:04:03 -07:00
Jason Lowe
85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Jason Lowe
14fec04ed3 Revert "YARN-6570. No logs were found for running application, running" 
This reverts commit 7c732924a8.
 2017-09-26 10:38:03 -05:00
Arun Suresh
df800f6cf3 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh) 2017-09-25 16:02:55 -07:00
Jason Lowe
bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Junping Du
e5e1851d80 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:01:16 -07:00
Jason Lowe
10d7493587 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 2017-09-21 08:38:08 -05:00
Junping Du
647b7527a9 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 2017-09-19 18:31:15 -07:00
Xuan
7c732924a8 YARN-6570. No logs were found for running application, running 
container. Contributed by Junping Du
 2017-09-18 14:04:05 -07:00
Jason Lowe
a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Arun Suresh
66ca0a6540 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
864fbacd45 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh) 2017-09-14 08:48:39 -07:00
Eric Payne
3860be7961 YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe. 2017-09-13 14:41:55 -05:00
Eric Payne
a3c44195be Revert 'YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.' 
This reverts commit 729d05f529.
 2017-09-13 14:38:58 -05:00
Eric Payne
729d05f529 YARN-4727. Unable to override the /home/ericp/run/conf/ env variable for container. Contributed by Jason Lowe. 2017-09-13 14:29:08 -05:00
Arun Suresh
a1c398e0dc YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix. 2017-09-12 17:03:32 -07:00
Arun Suresh
2ae72692fc YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh) 2017-09-12 16:10:08 -07:00
Wangda Tan
e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Varun Vasudev
759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Yufei Gu
5ab21dfe9c YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu) 2017-09-10 23:17:59 -07:00
Junping Du
3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
Arun Suresh
1f53ae7972 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 2017-09-08 09:24:05 -07:00
Jason Lowe
b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Wangda Tan
f155ab7cfa YARN-7033. Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan) 
Change-Id: Iffd18bb95debe1c8cc55e30abc1d8f663e9d0e30
 2017-09-07 14:13:37 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Allen Wittenauer
0adc3a0533
YARN-6721. container-executor should have stack checking 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-31 19:39:31 -07:00
Junping Du
91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Allen Wittenauer
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Jian He
08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Haibo Chen
9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Rohith Sharma K S
32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jian He
b664569586 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Haibo Chen
a8f082a180 YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 47474fffac)
 2017-08-30 11:29:52 +05:30
Sangjin Lee
092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Varun Saxena
c92a7ab31c YARN-3359. Addendum for Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30