big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,265 Commits 2 Branches 2 Tags 264 MiB
d0e21ac16b
Commit Graph

1161 Commits

Author SHA1 Message Date
Rohith Sharma K S
9119b3cf8f YARN-8155. Improve ATSv2 client logging in RM and NM publisher. Contributed by Abhishek Modi. 2018-06-14 12:38:10 +05:30
Eric Yang
22994889dc YARN-8259. Improve privileged docker container liveliness checks. 
Contributed by Shane Kumpf
 2018-06-13 19:24:31 -04:00
Inigo Goiri
6e756e8a62 HADOOP-15529. ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola. 2018-06-12 10:24:34 -07:00
Akira Ajisaka
652bcbb3e4
YARN-8363. Upgrade commons-lang version to 3.7 in hadoop-yarn-project. Contributed by Takanobu Asanuma. 2018-06-13 00:35:37 +09:00
Inigo Goiri
2b2f672022 YARN-8370. Some Node Manager tests fail on Windows due to improper path/file separator. Contributed by Anbang Hu. 2018-06-11 19:25:56 -07:00
Inigo Goiri
3b88fe25ba YARN-8359. Exclude containermanager.linux test classes on Windows. Contributed by Jason Lowe. 2018-06-07 17:13:03 -07:00
Robert Kanter
351cf87c92 Disable mounting cgroups by default (miklos.szegedi@cloudera.com via rkanter) 2018-06-07 17:09:34 -07:00
Miklos Szegedi
d5eca1a6a0 YARN-6677. Preempt opportunistic containers when root container cgroup goes over memory limit. Contributed by Haibo Chen. 2018-06-07 16:38:23 -07:00
Rohith Sharma K S
58bc34f1e3 YARN-8399. NodeManager is giving 403 GSS exception post upgrade to 3.1 in secure mode. Contributed by Sunil Govindan. 2018-06-07 10:25:56 +05:30
Haibo Chen
04cf699dd5 YARN-8388. TestCGroupElasticMemoryController.testNormalExit() hangs on Linux. (Miklos Szegedi via Haibo Chen) 2018-06-04 14:23:08 -07:00
Miklos Szegedi
e2c172dc9f YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian. 2018-06-04 10:41:10 -07:00
Billie Rinaldi
31998643a5 YARN-8342. Enable untrusted docker image to run with launch command. Contributed by Eric Yang 2018-06-02 14:46:32 -07:00
Haibo Chen
4880d890ee YARN-8375. TestCGroupElasticMemoryController fails surefire build. (Miklos Szegedi via Haibo Chen) 2018-06-01 16:29:13 -07:00
Wangda Tan
3a6bd77550 YARN-8384. stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop. (Eric Yang via wangda) 
Change-Id: Idbb2cd250f4f62dc32993e0d3ca3ec5684616baa
 2018-06-01 14:49:17 -07:00
Eric E Payne
e44c0849d7 YARN-8377: Javadoc build failed in hadoop-yarn-server-nodemanager. Contributed by Takanobu Asanuma 2018-05-30 16:50:19 +00:00
Eric Yang
135941e00d YARN-8362. Bugfix logic in container retries in node manager. 
Contributed by Chandni Singh
 2018-05-29 16:56:58 -04:00
Jason Lowe
4827e9a908 YARN-8329. Docker client configuration can still be set incorrectly. Contributed by Shane Kumpf 2018-05-29 14:43:17 -05:00
Shane Kumpf
1388de18ad YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:30:39 -06:00
Rohith Sharma K S
c05b5d424b YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 2018-05-24 14:19:46 +05:30
Haibo Chen
d996479954 YARN-4599. Set OOM control for memory cgroups. (Miklos Szegedi via Haibo Chen) 2018-05-23 16:35:37 -07:00
Inigo Goiri
e99e5bf104 YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola. 2018-05-23 14:15:26 -07:00
Shane Kumpf
9837ca9cc7 YARN-8285. Remove unused environment variables from the Docker runtime. Contributed by Eric Badger 2018-05-23 06:43:44 -06:00
Robert Kanter
b22f56c471 YARN-8273. Log aggregation does not warn if HDFS quota in target directory is exceeded (grepas via rkanter) 2018-05-22 14:24:38 -07:00
Eric Yang
6176d2b35c YARN-7960. Added security flag no-new-privileges for YARN Docker integration. 
Contributed by Eric Badger
 2018-05-22 13:44:58 -04:00
Jason Lowe
5f11288e41 YARN-8206. Sending a kill does not immediately kill docker containers. Contributed by Eric Badger 2018-05-22 09:27:08 -05:00
Arun Suresh
3159bffce2 YARN-7900. [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor. (Botong Huang via asuresh) 2018-05-17 20:00:52 -07:00
Eric Yang
d45a0b7d73 YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag. 
Contributed by Chandni Singh
 2018-05-17 17:29:34 -04:00
Jason Lowe
be53969047 YARN-8071. Add ability to specify nodemanager environment variables individually. Contributed by Jim Brennan 2018-05-16 16:17:28 -05:00
Jason Lowe
d47c09dcb1 YARN-8284. get_docker_command refactoring. Contributed by Eric Badger 2018-05-16 09:23:49 -05:00
Haibo Chen
2d00a0c71b YARN-8130 Race condition when container events are published for KILLED applications. (Rohith Sharma K S via Haibo Chen) 2018-05-14 11:08:42 -07:00
Eric Yang
0ff94563b9 YARN-8265. Improve DNS handling on docker IP changes. 
Contributed by Billie Rinaldi
 2018-05-11 22:37:43 -07:00
Jason Lowe
6c8e51ca7e YARN-7654. Support ENTRY_POINT for docker container. Contributed by Eric Yang 2018-05-11 18:56:05 -05:00
Jason Lowe
dc912994a1 YARN-8244. TestContainerSchedulerQueuing.testStartMultipleContainers failed. Contributed by Jim Brennan 2018-05-11 14:07:32 -05:00
Eric Yang
8f7912e0fe YARN-8274. Fixed a bug on docker start command. 
Contributed by Jason Lowe
 2018-05-11 14:23:16 -04:00
Haibo Chen
6341c3a437 YARN-7715. Support NM promotion/demotion of running containers. (Miklos Szegedi via Haibo Chen) 2018-05-10 11:01:01 -07:00
Eric Yang
af4fc2e628 YARN-8261. Fixed a bug in creation of localized container directory. 
Contributed by Jason Lowe
 2018-05-09 13:32:51 -04:00
Jason Lowe
a2ea756420 YARN-8207. Docker container launch use popen have risk of shell expansion. Contributed by Eric Yang. 2018-05-08 15:30:36 -05:00
Eric Yang
8cdb032aff YARN-8223. Improved yarn auxiliary service to load jar file from HDFS. 
Contributed by Zian Chen
 2018-05-04 12:36:31 -04:00
Billie Rinaldi
502914ca32 YARN-7818. Remove privileged operation warnings during container launch for the ContainerRuntimes. Contributed by Shane Kumpf 2018-05-04 08:53:55 -07:00
Eric Yang
f4d280f02b YARN-8194. Fixed reinitialization error for LinuxContainerExecutor. 
Contributed by Chandni Singh
 2018-05-02 20:07:19 -04:00
Eric Yang
883f68222a YARN-8209. Fixed NPE in Yarn Service deletion. 
Contributed by Eric Badger
 2018-05-02 14:33:31 -04:00
Eric Yang
a966ec6e23 YARN-8228. Added hostname length check for docker container. 
Contributed by Shane Kumpf
 2018-04-30 19:12:53 -04:00
Ewan Higgs
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-27 19:21:37 +09:00
Sunil G
7fc09c4f72 YARN-8215. ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM. Contributed by Rohith Sharma K S. 2018-04-27 10:33:42 +05:30
Eric Yang
85585f9eef YARN-8064. Bugfix for clean up container-executor cmd helper file. 
Contributed by Eric Badger
 2018-04-20 13:49:33 -04:00
Haibo Chen
860cc28a5a YARN-8185. Improve log in DirectoryCollection constructor. (Yufei Gu via Haibo Chen) 2018-04-20 07:41:24 -07:00
Shashikant Banerjee
e4c39f3247
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-19 08:35:38 +09:00
Haibo Chen
49f9acae40 YARN-8162. Remove Method DirectoryCollection#verifyDirUsingMkdir. (Yufei Gu via Haibo Chen) 2018-04-16 15:05:04 -07:00
Weiwei Yang
226bedc023 YARN-8153. Guaranteed containers always stay in SCHEDULED on NM after restart. Contributed by Yang Wang. 2018-04-13 13:17:37 +08:00
Eric E Payne
6bb128dfb8 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 2018-04-12 16:04:23 +00:00
Billie Rinaldi
933477e9e0 YARN-7221. Add security check for privileged docker container. Contributed by Eric Yang 2018-04-11 11:24:23 -07:00
Wangda Tan
2bf9cc2c73 YARN-8116. Nodemanager fails with NumberFormatException: For input string: . (Chandni Singh via wangda) 
Change-Id: Idd30cfca59982d3fc6e47aa1b88f844a78fae94d
 2018-04-10 17:32:38 -07:00
Eric Yang
c467f311d0 YARN-7973. Added ContainerRelaunch feature for Docker containers. 
Contributed by Shane Kumpf
 2018-04-10 19:25:00 -04:00
Jason Lowe
907919d28c YARN-7667. Docker Stop grace period should be configurable. Contributed by Eric Badger 2018-04-09 17:19:21 -05:00
bibinchundatt
eb47c3de74 YARN-7905. Parent directory permission incorrect during public localization. Contributed by Bilwa S T. 2018-04-07 12:26:29 +05:30
Wangda Tan
00ebec89f1 YARN-1151. Ability to configure auxiliary services from HDFS-based JAR files. (Xuan Gong via wangda) 
Change-Id: Ied37ff11e507fc86847753ba79486652c8fadfe9
 2018-04-06 21:25:57 -07:00
Subru Krishnan
00905efab2 YARN-8110. AMRMProxy recover should catch for all throwable to avoid premature exit. (Botong Huang via Subru). 2018-04-06 16:31:16 -07:00
Miklos Szegedi
2d06d885c8 YARN-8035. Uncaught exception in ContainersMonitorImpl during relaunch due to the process ID changing. Contributed by Shane Kumpf. 2018-04-03 10:01:00 -07:00
Jason Lowe
54a81211ea YARN-8082. Include LocalizedResource size information in the NM download log for localization. Contributed by Kuhu Shukla 2018-04-02 10:38:13 -05:00
Miklos Szegedi
cdee0a4f84 YARN-8077. The vmemLimit parameter in ContainersMonitorImpl#isProcessTreeOverLimit is confusing. Contributed by Sen Zhao. 2018-03-28 09:05:25 -07:00
Weiwei Yang
411993f6e5 YARN-7734. Fix UT failure TestContainerLogsPage#testContainerLogPageAccess. Contributed by Tao Yang. 2018-03-28 17:35:38 +08:00
Eric Yang
e6b74164e9 YARN-8043. Added the exception message for failed launches running under LCE. 
Contributed by Shane Kumpf
 2018-03-26 13:21:35 -04:00
Jason Lowe
5aa7052e31 YARN-8054. Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread. Contributed by Jonathan Eagles 2018-03-21 17:19:20 -05:00
Jason Lowe
f480367af6 YARN-8027. Setting hostname of docker container breaks for --net=host in docker 1.13. Contributed by Jim Brennan 2018-03-19 16:59:58 -05:00
Yufei Gu
49c747ab18 YARN-8039. Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizer. Contributed by Miklos Szegedi. 2018-03-16 16:25:33 -07:00
Wangda Tan
4bf622043f YARN-7952. RM should be able to recover log aggregation status after restart/fail-over. (Xuan Gong via wangda) 
Change-Id: I725c9afe64831eda0aa6b0bebdbc79d2dd165707
 2018-03-15 13:26:45 -07:00
fang zhenyi
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Eric Yang
a82be7754d YARN-7999. Added file cache initializer for Linux container-executor. 
Contributed by Jason Lowe
 2018-03-14 12:21:57 -04:00
Wangda Tan
a5b27b3c67 YARN-5015. Support sliding window retry capability for container restart. (Chandni Singh via wangda) 
Change-Id: I07addd3e4ba8d98456ee2ff1d5c540a38fe61dea
 2018-03-13 17:55:17 -07:00
Miklos Szegedi
a82d4a2e3a YARN-5764. NUMA awareness support for launching containers. Contributed by Devaraj K. 2018-03-13 12:36:57 -07:00
Wangda Tan
e718ac597f Revert "YARN-7891. LogAggregationIndexedFileController should support read from HAR file. (Xuan Gong via wangda)" 
This reverts commit 4d53ef7eef.
 2018-03-07 15:46:47 -08:00
Wangda Tan
037d783483 YARN-7626. Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount. (Zian Chen via wangda) 
Change-Id: If461277d4557922ab7e4dce9dd8dc5d0d5f22710
(cherry picked from commit 88f9138e12d2d5a1bd13f0915acef93037c1d086)
 2018-03-07 11:30:15 -08:00
Wangda Tan
4d53ef7eef YARN-7891. LogAggregationIndexedFileController should support read from HAR file. (Xuan Gong via wangda) 
Change-Id: Ie16e34039d57df50128c73b37516ad0bc7c9590e
 2018-03-07 11:30:06 -08:00
Jason Lowe
d69b31f7f7 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-03-07 09:20:34 -06:00
Billie Rinaldi
628be58a4c YARN-7915. Trusted image log message repeated multiple times. Contributed by Shane Kumpf 2018-03-05 09:08:44 -08:00
Jason Lowe
83798f15f8 YARN-7511. NPE in ContainerLocalizer when localization failed for running container. Contributed by Tao Yang 2018-03-02 09:59:46 -06:00
Ray Chiang
923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
Sunil G
22928c0c60 YARN-7637. GPU volume creation command fails when work preserving is disabled at NM. Contributed by Zian Chen. 2018-03-01 22:39:32 +05:30
Billie Rinaldi
727c033997 YARN-7446. Remove --user flag when running privileged mode docker container. Contributed by Eric Yang 2018-02-27 14:33:57 -08:00
Jason Lowe
8e728f39c9 YARN-5714. ContainerExecutor does not order environment map. Contributed by Remi Catherinot and Jim Brennan 2018-02-23 15:46:35 -06:00
Jason Lowe
3132709b46 YARN-7916. Remove call to docker logs on failure in container-executor. Contributed by Shane Kumpf 2018-02-21 16:54:02 -06:00
Jason Lowe
b9a429bb28 Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 8013475d44.
 2018-02-19 08:16:25 -06:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Jason Lowe
8013475d44 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-02-15 17:09:00 -06:00
Jason Lowe
5a1db60ab1 YARN-7914. Fix exit code handling for short lived Docker containers. Contributed by Shane Kumpf 2018-02-12 15:50:10 -06:00
Jason Lowe
3414fd1243 YARN-7917. Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens. Contributed by Shane Kumpf 2018-02-12 09:27:43 -06:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Jian He
eb2449d539 YARN-5428. Allow for specifying the docker client configuration directory. Contributed by Shane Kumpf 2018-02-08 11:35:30 -08:00
Jason Lowe
456705a07c YARN-7815. Make the YARN mounts added to Docker containers more restrictive. Contributed by Shane Kumpf 2018-02-07 13:09:08 -06:00
Billie Rinaldi
266da25c04 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit e14f03dfbf.
 2018-02-07 07:17:00 -08:00
Billie Rinaldi
aa461f9091 YARN-7516. Add security check for trusted docker images. Contributed by Eric Yang 2018-02-07 07:09:45 -08:00
Anu Engineer
4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Billie Rinaldi
a196ee9362 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 2018-02-05 18:04:09 -08:00
Jason Lowe
682ea21f2b Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 12eaae383a.
 2018-02-01 12:14:09 -06:00
Jason Lowe
12eaae383a YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-01-31 11:37:54 -06:00
Arun Suresh
a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
6463e10c72 YARN-7723. Avoid using docker volume --format option to run against to older docker releases. Contributed by Wangda Tan 2018-01-30 15:58:11 +05:30
Jian He
443523f9c0 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:18:51 -08:00
Haibo Chen
649ef7ac33 YARN-7064. Use cgroup to get container resource utilization. (Miklos Szegedi via Haibo Chen) 2018-01-26 16:27:31 -08:00
Billie Rinaldi
f2fa736f0a YARN-7797. Docker host network can not obtain IP address for RegistryDNS. Contributed by Eric Yang 2018-01-26 09:32:23 -08:00
Miklos Szegedi
2e5865606b YARN-7814. Remove automatic mounting of the cgroups root directory into Docker containers. Contributed by Shane Kumpf. 2018-01-25 17:37:23 -08:00
Eric Yang
59828be197 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)
 2018-01-24 21:24:19 -05:00
Zhe Zhang
fa8cf4d1b4 YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 2018-01-24 12:54:54 -08:00
Miklos Szegedi
e7642a3e6f YARN-7796. Container-executor fails with segfault on certain OS configurations. Contributed by Gergo Repas. 2018-01-23 21:07:05 -08:00
Eric Yang
97fe3cc187 YARN-7729. Add support for setting Docker PID namespace mode. (Contributed by Billie Rinaldi) 2018-01-22 16:33:38 -05:00
Haibo Chen
4aca4ff759 YARN-5094. some YARN container events have timestamp of -1. 2018-01-19 10:27:46 -08:00
Billie Rinaldi
37f4696a9c YARN-7740. Fix logging for destroy yarn service cli when app does not exist and some minor bugs. Contributed by Jian He 2018-01-18 12:11:19 -08:00
Miklos Szegedi
a68e445dc6 YARN-7717. Add configuration consistency for module.enabled and docker.privileged-containers.enabled. Contributed by Eric Badger. 2018-01-17 14:11:14 -08:00
Miklos Szegedi
41049ba5d1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-16 15:40:43 -08:00
Miklos Szegedi
2dcfc1876e YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 2018-01-12 17:00:01 -08:00
Wangda Tan
edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Eric Yang
3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Miklos Szegedi
bc285da107 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 2018-01-11 09:40:09 -08:00
Jason Lowe
d795661868 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:15:52 -06:00
Haibo Chen
2f6c038be6 YARN-7602. NM should reference the singleton JvmMetrics instance. 2018-01-03 09:41:26 -08:00
Arun Suresh
c0c7cce81d YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 2018-01-02 22:03:00 -08:00
Miklos Szegedi
7f515f57ed YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 2018-01-02 18:03:04 -08:00
Miklos Szegedi
b82049b4f0 YARN-7580. ContainersMonitorImpl logged message lacks detail when exceeding memory limits. Contributed by Wilfred Spiegelenburg. 2017-12-29 12:49:37 -08:00
Arun Suresh
a55884c68e YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 2017-12-28 22:20:42 -08:00
Jason Lowe
811fabdebe YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 2017-12-18 15:20:06 -06:00
Akira Ajisaka
001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Wangda Tan
44825f0960 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
 2017-12-15 15:40:56 -08:00
Wangda Tan
631b5c2db7 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
 2017-12-15 15:38:36 -08:00
Chen Liang
46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Weiwei Yang
7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe
2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jason Lowe
06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Subru Krishnan
670e8d4ec7 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 2017-12-08 15:39:18 -08:00
Wangda Tan
04b84da245 YARN-7443. Add native FPGA module support to do isolation with cgroups. (Zhankun Tang via wangda) 
Change-Id: Ic4b7f9f3e032986b8f955139c9fe4d3a6c818a53
 2017-12-08 15:18:22 -08:00
Weiwei Yang
e411dd6666 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:29:40 +08:00
Weiwei Yang
05c347fe51 YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 2017-12-06 12:21:52 +08:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Sunil G
81f6e46b2f YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 2017-12-04 11:27:23 +05:30
Sunil G
30f2646b15 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2017-12-04 10:45:07 +05:30
Jason Lowe
60f95fb719 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 2017-12-01 15:47:01 -06:00
Wangda Tan
7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Sunil G
556aea3f36 YARN-7487. Ensure volume to include GPU base libraries after created by plugin. Contributed by Wangda Tan. 2017-12-01 13:36:28 +05:30
Wangda Tan
c9a54aab6b YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2017-11-29 17:46:16 -08:00
Yufei Gu
d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Sunil G
aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Eric Yang
d42a336cfa YARN-5534. Allow user provided Docker volume mount list. (Contributed by Shane Kumpf) 2017-11-22 13:05:34 -05:00
Subru Krishnan
d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Junping Du
e14f03dfbf YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 15:27:37 -08:00
Arun Suresh
cb35a59589 YARN-7343. Add a junit test for ContainerScheduler recovery. (Sampada Dehankar via asuresh) 2017-11-08 08:14:02 -08:00
Jian He
a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Billie Rinaldi
ce74e64363 YARN-7210. Some NPE fixes in Registry DNS. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Jian He
8d335e59cf YARN-6804. [yarn-native-services changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-11-06 13:30:10 -08:00
Billie Rinaldi
ce05c6e981 YARN-6545. Followup fix for YARN-6405. Contributed by Jian He 2017-11-06 13:30:07 -08:00
bibinchundatt
dcd99c4b9a Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 2017-11-06 22:39:10 +05:30
Jason Lowe
d00b6f7c1f YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 2017-11-02 09:37:17 -05:00
Jian He
0cc98ae0ec YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2017-11-01 17:00:32 -07:00
Eric Yang
7a49ddfdde YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:39:56 -04:00
Sunil G
9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Arun Suresh
9c5c68745e YARN-7299. Fix TestDistributedScheduler. (asuresh) 2017-10-27 23:08:18 -07:00
Jason Lowe
665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Haibo Chen
480187aebb YARN-7372. TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic is flaky. 2017-10-20 14:24:17 -07:00
Yufei Gu
0799fde35e YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 2017-10-20 10:00:13 -07:00
Eric Yang
b61144a93d YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-20 12:02:06 -04:00
Wangda Tan
c1b08ba720 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2017-10-19 14:45:44 -07:00
Nathan Roberts
4540ffd15f YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe. 2017-10-16 17:00:38 -05:00
Arun Suresh
a50be1b8f4 YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh) 2017-10-16 13:12:15 -07:00
Sunil G
fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Haibo Chen
c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Naganarasimha
d0046bb5c6 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 2017-10-06 00:07:53 +05:30
Sunil G
592bf2d550 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton. 2017-10-05 10:52:00 +05:30
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Sidharta S
7eb8499996 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-02 19:04:49 -07:00
Wangda Tan
febeead5f9 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
 2017-09-29 16:39:46 -07:00
Wangda Tan
d3b1c63195 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
 2017-09-28 16:41:09 -07:00
Subru Krishnan
ca669f9f8b YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 2017-09-28 13:04:03 -07:00
Jason Lowe
85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Jason Lowe
14fec04ed3 Revert "YARN-6570. No logs were found for running application, running" 
This reverts commit 7c732924a8.
 2017-09-26 10:38:03 -05:00
Arun Suresh
df800f6cf3 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh) 2017-09-25 16:02:55 -07:00
Jason Lowe
bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Junping Du
e5e1851d80 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:01:16 -07:00
Jason Lowe
10d7493587 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 2017-09-21 08:38:08 -05:00
Junping Du
647b7527a9 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 2017-09-19 18:31:15 -07:00
Xuan
7c732924a8 YARN-6570. No logs were found for running application, running 
container. Contributed by Junping Du
 2017-09-18 14:04:05 -07:00
Jason Lowe
a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Arun Suresh
66ca0a6540 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh
864fbacd45 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh) 2017-09-14 08:48:39 -07:00
Eric Payne
3860be7961 YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe. 2017-09-13 14:41:55 -05:00
Eric Payne
a3c44195be Revert 'YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.' 
This reverts commit 729d05f529.
 2017-09-13 14:38:58 -05:00
Eric Payne
729d05f529 YARN-4727. Unable to override the /home/ericp/run/conf/ env variable for container. Contributed by Jason Lowe. 2017-09-13 14:29:08 -05:00
Arun Suresh
a1c398e0dc YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix. 2017-09-12 17:03:32 -07:00
Arun Suresh
2ae72692fc YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh) 2017-09-12 16:10:08 -07:00
Wangda Tan
e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Varun Vasudev
759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Yufei Gu
5ab21dfe9c YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu) 2017-09-10 23:17:59 -07:00
Junping Du
3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
Arun Suresh
1f53ae7972 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 2017-09-08 09:24:05 -07:00
Jason Lowe
b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Wangda Tan
f155ab7cfa YARN-7033. Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan) 
Change-Id: Iffd18bb95debe1c8cc55e30abc1d8f663e9d0e30
 2017-09-07 14:13:37 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Allen Wittenauer
0adc3a0533
YARN-6721. container-executor should have stack checking 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-31 19:39:31 -07:00
Junping Du
91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Allen Wittenauer
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Jian He
08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30