big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,120 Commits 2 Branches 2 Tags 264 MiB
dd72e8fbc3
Commit Graph

1408 Commits

Author SHA1 Message Date
Szilard Nemeth
31f172fd96 YARN-9677. Make FpgaDevice and GpuDevice classes more similar to each other. Contributed by kevin su 2019-11-09 14:04:14 +01:00
Eric Badger
eef34f2d87 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 2019-10-25 17:13:04 +00:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Eric Yang
72b1bed998 YARN-9884. Make container-executor mount logic modular 
Contributed by Eric Badger
 2019-10-18 19:10:32 -04:00
Eric Yang
31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN 
Contributed by Prabhu Joseph and Shane Kumpf
 2019-10-10 19:02:02 -04:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Sunil G
c474e24c0b YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 2019-09-17 19:41:38 +05:30
HUAN-PING SU
4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
 2019-09-13 10:36:45 -07:00
Abhishek Modi
b939ae98e6 YARN-8174. Add containerId to ResourceLocalizationService fetch failure log statement. Contributed by Prabhu Joseph. 2019-09-02 10:45:59 +05:30
Szilard Nemeth
e8fa192f07 YARN-9217. Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing. Contributed by Peter Bacsko 2019-08-21 16:44:22 +02:00
bibinchundatt
e684b17e6f YARN-5857. TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk. Contributed by Bilwa S T. 2019-08-21 17:14:42 +05:30
Szilard Nemeth
2216ec54e5 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 09:13:20 +02:00
Szilard Nemeth
2a05e0ff3b YARN-9749. TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk. Contributed by Adam Antal 2019-08-16 08:52:09 +02:00
Adam Antal
22c4f38c4b YARN-9679. Regular code cleanup in TestResourcePluginManager (#1122) 2019-08-15 17:32:05 +02:00
HUAN-PING SU
167acd87da YARN-9683. Remove reapDockerContainerNoPid left behind by YARN-9074 (#1212) Contributed by Kevin Su. 
Reviewed-by: Eric Yang <eyang@apache.org>
Reviewed-by: Adam Antal <adam.antal@cloudera.com>
 2019-08-14 10:42:29 -07:00
Adam Antal
c89bdfacc8 YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… (#1261) 
* YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes

* Using {} placeholder, and increasing loglevel if log aggregation failed.
 2019-08-14 17:35:16 +02:00
Szilard Nemeth
3e0410449f YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. Contributed by Peter Bacsko 2019-08-14 17:13:54 +02:00
Szilard Nemeth
e5e609384f YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 16:58:22 +02:00
Eric Badger
2ac029b949 YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 2019-08-13 16:21:18 +00:00
Eric Yang
6ff0453ede YARN-9527. Prevent rogue Localizer Runner from downloading same file repeatly. 
Contributed by Jim Brennan
 2019-08-09 14:12:17 -04:00
Szilard Nemeth
e0c21c6da9 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak 2019-08-09 10:18:34 +02:00
Szilard Nemeth
742e30b473 YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak 2019-08-09 09:59:19 +02:00
Szilard Nemeth
72d7e570a7 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak 2019-08-09 09:49:18 +02:00
Haibo Chen
f51702d539 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 2019-08-06 13:52:30 -07:00
HUAN-PING SU
7c2042a44d YARN-9678. Addendum: TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 10:21:55 -07:00
HUAN-PING SU
b8bf09ba3d YARN-9678. TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 09:05:53 -07:00
Eric Yang
d6697da5e8 YARN-9667. Use setbuf with line buffer to reduce fflush complexity in container-executor. 
Contributed by Peter Bacsko
 2019-08-05 13:59:12 -04:00
Szilard Nemeth
54ac80176e Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph 2019-08-02 13:38:06 +02:00
Vidura Mudalige
1930a7bf60 YARN-9093. Remove commented code block from the beginning of Tes… (#444) 2019-08-02 13:16:19 +02:00
Adam Antal
95fc38f2e9 YARN-9375. Use Configured in GpuDiscoverer and FpgaDiscoverer (#1131) 
Contributed by Adam Antal
 2019-08-02 11:24:09 +02:00
Szilard Nemeth
18ee1092b4 YARN-9127. Create more tests to verify GpuDeviceInformationParser. Contributed by Peter Bacsko 2019-07-15 11:59:11 +02:00
Szilard Nemeth
61b0c2bb7c YARN-9337. GPU auto-discovery script runs even when the resource is given by hand. Contributed by Adam Antal 2019-07-12 17:28:14 +02:00
Szilard Nemeth
8b3c6791b1 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-07-12 17:20:42 +02:00
Szilard Nemeth
c416284bb7 YARN-9235. If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown. Contributed by Antal Balint Steinbach, Adam Antal 2019-07-12 16:51:58 +02:00
Haibo Chen
9b54dd7186 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonathan Hung) 2019-07-11 13:57:08 -07:00
Szilard Nemeth
a2a8be18cb YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by Adam Antal. 2019-07-03 13:45:00 +02:00
Weiwei Yang
570eee30e5 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi. 2019-07-02 09:55:25 +08:00
Eric Yang
29465bf169 YARN-9560. Restructure DockerLinuxContainerRuntime to extend OCIContainerRuntime. 
Contributed by Eric Badger, Jim Brennan, Craig Condit
 2019-06-28 17:18:53 -04:00
Weiwei Yang
f09c31a97e Revert "YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023" 
This reverts commit 5e7caf1287.
 2019-06-29 00:29:17 +08:00
Weiwei Yang
5e7caf1287 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023 2019-06-29 00:08:40 +08:00
Giovanni Matteo Fumarola
1ac967a6b7 YARN-6055. ContainersMonitorImpl need be adjusted when NM resource changed. Contributed by Inigo Goiri. 2019-06-26 14:01:31 -07:00
Zhankun Tang
062eb605ac YARN-9477. Implement VE discovery using libudev. Contributed by Peter Bacsko. 2019-06-26 23:53:14 +08:00
Zhankun Tang
67414a1a80 YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 2019-06-18 12:23:52 +08:00
Giovanni Matteo Fumarola
bcfd228336 YARN-9599. TestContainerSchedulerQueuing#testQueueShedding fails intermittently. Contributed by Abhishek Modi. 2019-06-13 11:08:35 -07:00
bibinchundatt
2263ead365 YARN-9557. Application fails in diskchecker when ReadWriteDiskValidator is configured. Contributed by Bilwa S T. 2019-06-11 23:20:28 +05:30
bibinchundatt
6d80b9bc3f YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 2019-06-11 22:49:21 +05:30
bibinchundatt
f7df55f4a8 YARN-9602. Use logger format in Container Executor. Contributed by Abhishek Modi. 2019-06-11 22:29:00 +05:30
Zhankun Tang
606061aa14 YARN-9595. FPGA plugin: NullPointerException in FpgaNodeResourceUpdateHandler.updateConfiguredResource(). Contributed by Peter Bacsko. 2019-06-04 09:56:59 +08:00
Weiwei Yang
4530f4500d YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T. 2019-06-03 14:09:37 +08:00
Giovanni Matteo Fumarola
2210897609 YARN-9592. Use Logger format in ContainersMonitorImpl. Contributed by Inigo Goiri. 2019-05-31 17:35:49 -07:00
Sunil G
7861a5eb1a YARN-9033. ResourceHandlerChain#bootstrap is invoked twice during NM start if LinuxContainerExecutor enabled. Contributed by Zhankun Tang. 2019-05-31 10:22:26 +05:30
Sunil G
30c6dd92e1 YARN-9452. Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures. Contributed by Prabhu Joseph. 2019-05-30 22:32:41 +05:30
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Eric Yang
460ba7fb14 YARN-9558. Fixed LogAggregation test cases. 
Contributed by Prabhu Joseph
 2019-05-23 18:38:47 -04:00
Haibo Chen
597fa47ad1 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 2019-05-06 11:56:22 -07:00
Eric Yang
accb811e57 YARN-6929. Improved partition algorithm for yarn remote-app-log-dir. 
Contributed by Prabhu Joseph
 2019-04-30 17:04:59 -04:00
Zhankun Tang
7fbaa7d66f YARN-9476. [YARN-9473] Create unit tests for VE plugin. Contributed by Peter Bacsko. 2019-04-30 11:06:44 +08:00
Eric Badger
79d3d35398 YARN-9486. Docker container exited with failure does not get clean up correctly. Contributed by Eric Yang 2019-04-26 01:21:28 +00:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Zhankun Tang
8a95ea61e1 YARN-9475. [YARN-9473] Create basic VE plugin. Contributed by Peter Bacsko. 2019-04-23 17:33:58 +08:00
Inigo Goiri
96e3027e46 YARN-2889. Limit the number of opportunistic container allocated per AM heartbeat. Contributed by Abhishek Modi. 2019-04-22 09:49:03 -07:00
Eric Yang
ef97a20831 YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng
 2019-04-18 18:59:21 -04:00
Prabhu Joseph
aa4c744aef
YARN-9470. Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-18 15:40:37 +09:00
Siyao Meng
6e4399ea61 YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-17 22:56:57 -07:00
Eric Yang
9cf7401794 YARN-9349. Improved log level practices for InvalidStateTransitionException. 
Contributed by Anuhan Torgonshar

(cherry picked from commit fe2370e039e1ee980d74769ae85d67434e0993cf)
 2019-04-16 19:53:45 -04:00
Szilard Nemeth
b8086aed86 YARN-9123. Clean up and split testcases in TestNMWebServices for GPU support. Contributed by Szilard Nemeth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-16 11:06:25 -07:00
Eric Badger
5583e1b6fc YARN-7848 Force removal of docker containers that do not get removed on first try. Contributed by Eric Yang 2019-04-15 20:47:09 +00:00
Eric Badger
254efc9358 YARN-9379. Can't specify docker runtime through environment. Contributed by caozhiqiang 2019-04-15 18:24:37 +00:00
Vrushali C
27039a29ae YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. 2019-04-05 12:02:43 -07:00
Giovanni Matteo Fumarola
ab2bda57bd YARN-9428. Add metrics for paused containers in NodeManager. Contributed by Abhishek Modi. 2019-04-01 14:21:17 -07:00
Giovanni Matteo Fumarola
332cab5518 YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. 2019-04-01 11:06:51 -07:00
Devaraj K
56f1e131ec YARN-9270. Minor cleanup in TestFpgaDiscoverer. Contributed by Peter Bacsko. 2019-03-29 10:58:56 -07:00
Devaraj K
a4cd75e09c YARN-9269. Minor cleanup in FpgaResourceAllocator. Contributed by Peter Bacsko. 2019-03-27 10:08:07 -07:00
Devaraj K
eeda6891e4 YARN-9268. General improvements in FpgaDevice. Contributed by Peter Bacsko. 2019-03-25 13:22:53 -07:00
Eric Yang
3c45762a0b YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan
 2019-03-25 15:53:24 -04:00
Devaraj K
a99eb80659 YARN-9267. General improvements in FpgaResourceHandlerImpl. Contributed by Peter Bacsko. 2019-03-21 11:15:56 -07:00
Eric Yang
506502bb83 YARN-9370. Added logging for recovering assigned GPU devices. 
Contributed by Yesha Vora
 2019-03-20 19:12:19 -04:00
Eric Yang
f2b862cac6 YARN-9398. Fixed javadoc errors for FPGA related java files. 
Contributed by Peter Bacsko
 2019-03-20 15:45:37 -04:00
Eric Yang
09eabda314 YARN-9364. Remove commons-logging dependency from YARN. 
Contributed by Prabhu Joseph
 2019-03-18 19:58:42 -04:00
Eric Yang
5f6e225166 YARN-9363. Replaced debug logging with SLF4J parameterized log message. 
Contributed by Prabhu Joseph
 2019-03-18 13:57:18 -04:00
Eric Yang
2064ca015d YARN-9349. Changed logging to use slf4j api. 
Contributed by Prabhu Joseph
 2019-03-15 19:20:59 -04:00
Eric Badger
688b177fc6 YARN-8376. Separate white list for docker.trusted.registries and docker.privileged-container.registries. Contributed by Eric Yang 2019-03-14 19:39:00 +00:00
Sunil G
8e1539eca8 YARN-9266. General improvements in IntelFpgaOpenclPlugin. Contributed by Peter Bacsko. 2019-03-13 02:45:17 +05:30
Sunil G
de15a66d78 YARN-9265. FPGA plugin fails to recognize Intel Processing Accelerator Card. Contributed by Peter Bacsko. 2019-03-08 17:39:22 +05:30
Eric Yang
39b4a37e02 YARN-9341. Fixed enentrant lock usage in YARN project. 
Contributed by Prabhu Joseph
 2019-03-07 16:47:45 -05:00
Sunil G
46045c5cb3 YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-06 16:01:08 +05:30
Suma Shivaprasad
cab8529ecb YARN-7904. Privileged, trusted containers should be supported only in ENTRYPOINT mode. Contributed by Eric Yang. 2019-03-01 11:06:09 -08:00
Sunil G
dcaca19871 YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-01 19:24:35 +05:30
Eric Yang
fbc7bb315f YARN-9245. Added query docker image command ability to node manager. 
Contributed by Chandni Singh
 2019-02-27 14:57:24 -05:00
Weiwei Yang
c6ea28c480 YARN-9331. [YARN-8851] Fix a bug that lacking cgroup initialization when bootstrap DeviceResourceHandlerImpl. Contributed by Zhankun Tang. 2019-02-26 10:05:31 +08:00
Giovanni Matteo Fumarola
95372657fc YARN-9287. Consecutive StringBuilder append should be reuse. Contributed by Ayush Saxena. 2019-02-25 11:45:37 -08:00
Sunil G
5e91ebd91a YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth. 2019-02-25 11:30:46 +05:30
Sunil G
dddcfa4d9f YARN-8821. [YARN-8851] GPU hierarchy/topology scheduling support based on pluggable device framework. Contributed by Zhankun Tang. 2019-02-24 14:37:06 +05:30
bibinchundatt
ed13cf8406 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-22 21:01:34 +05:30
Sunil G
95fbbfed75 YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth. 2019-02-22 20:22:17 +05:30
Weiwei Yang
632d5e8a98 YARN-9156. Improve debug message in device plugin method compatibility check of ResourcePluginManager. Contributed by Zhankun Tang. 2019-02-22 11:27:30 +08:00
Wangda Tan
b17a2602d1 YARN-9319. Fix compilation issue of handling typedef an existing name by gcc compiler. (Zhankun Tang via wangda) 
Change-Id: Ie8293cc5b710ff355fb6a1ea7201be53ed0385bf
 2019-02-21 11:17:32 -08:00
Sunil G
db4d1a1e2f YARN-9060. [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example. Contributed by Zhankun Tang. 2019-02-18 15:58:04 +05:30
Eric Yang
7c1b561e33 YARN-8927. Added support for top level Dockerhub images to trusted registry using library keyword. 
Contributed by Zhankun Tang
 2019-02-15 11:18:07 -05:00
Adam Antal
9385ec45d7
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-02-15 18:47:10 +09:00
Eric Yang
3dc2523266 YARN-9184. Add a system flag to allow update to latest docker images. 
Contributed by Zhaohui Xin
 2019-02-12 16:16:35 -05:00
Masatake Iwasaki
e0ab1bdece YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 2019-02-09 00:26:50 +09:00
Vinayakumar B
3c96a03dea YARN-8498. Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04. Contributed by Ayush Saxena. 2019-02-07 13:02:35 +05:30
Rohith Sharma K S
e3ec18b0c4 YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi. 2019-02-05 13:48:04 +05:30
Weiwei Yang
f20b043a02 YARN-9263. TestConfigurationNodeAttributesProvider fails after Mockito updated. Contributed by Weiwei Yang. 2019-02-02 23:04:34 +08:00
Sunil G
71c49fa60f YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way. Contributed by Szilard Nemeth. 2019-01-31 09:25:29 +05:30
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Weiwei Yang
4f63ffe444 YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang. 2019-01-29 10:34:26 +08:00
Eric Yang
2e636dd3c4 YARN-9074. Consolidate docker removal logic in ContainerCleanup. 
Contributed by Zhaohui Xin
 2019-01-28 18:05:53 -05:00
Eric Yang
1ab69a9543 YARN-9221. Added flag to disable dynamic auxiliary service feature. 
Contributed by Billie Rinaldi
 2019-01-25 19:05:36 -05:00
Eric Yang
a33ef4fd31 YARN-8867. Added resource localization status to YARN service status call. 
Contributed by Chandni Singh
 2019-01-24 18:43:21 -05:00
Eric Yang
2fa9389c2e YARN-9146. Added REST API to configure auxiliary service. 
Contributed by Billie Rinaldi
 2019-01-22 18:24:43 -05:00
Jason Lowe
6a923464af YARN-6523. Optimize system credentials sent in node heartbeat responses. Contributed by Manikandan R 2019-01-08 16:54:05 -06:00
Weiwei Yang
0921b706f7 YARN-9037. [CSI] Ignore volume resource in resource calculators based on tags. Contributed by Sunil Govindan. 2019-01-08 14:30:53 +08:00
Giovanni Matteo Fumarola
489411579c YARN-9169. Add metrics for queued opportunistic and guaranteed containers. Contributed by Abhishek Modi. 2019-01-07 15:16:55 -08:00
Wangda Tan
0a01d49917 YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda) 
Change-Id: Id8af27134d3286a7a10d85eda9be25df9689d0e7
 2019-01-07 12:07:26 -08:00
Giovanni Matteo Fumarola
6e35f7130f YARN-9166. Fix logging for preemption of Opportunistic containers for Guaranteed containers. Contributed by Abhishek Modi. 2019-01-04 11:16:41 -08:00
Sunil G
f4906ac019 YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang. 2019-01-04 12:10:00 +05:30
Eric Yang
dfceffa70d YARN-9147. Rmove auxiliary services when manifest file is removed. 
Contributed by Billie Rinaldi
 2019-01-03 12:57:21 -05:00
Botong Huang
657aa433e2 YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi. 2018-12-22 12:41:49 -08:00
Eric Yang
f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Eric Yang
ea724181d6 YARN-9132. Added file permission check for auxiliary services manifest file. 
Contributed by Billie Rinaldi
 2018-12-21 14:56:39 -05:00
Weiwei Yang
f659485ee8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 10:56:42 +08:00
Eric Yang
a80d321074 YARN-9152. Improved AuxServices REST API output. 
Contributed by Billie Rinaldi
 2018-12-20 19:21:55 -05:00
Billie Rinaldi
2e544dc921 YARN-9129. Ensure flush after printing to log plus additional cleanup. Contributed by Eric Yang 2018-12-19 16:45:23 -08:00
Eric Yang
e815fd9c49 YARN-9126. Fix container clean up for reinitialization. 
Contributed by Chandni Singh
 2018-12-19 14:55:56 -05:00
Eric Yang
c7a5a4435e YARN-9075. Add ability to register/remove auxiliary service without restart node manager. 
Contributed by Billie Rinaldi
 2018-12-18 17:05:51 -05:00
Billie Rinaldi
a6e9d27c29 YARN-9117. Add a check for insecure setup for container terminal. Contributed by Eric Yang 2018-12-18 12:49:10 -08:00
Billie Rinaldi
c5c73182e5 YARN-9072. Send exit command to terminate docker exec on connection close. Contributed by Eric Yang 2018-12-18 10:06:33 -08:00
Billie Rinaldi
4f622337ee YARN-9091. Improve terminal message when connection is refused. Contributed by Eric Yang 2018-12-14 19:55:50 -08:00
Billie Rinaldi
42297f7d99 YARN-8963. Add flag to disable interactive shell. Contributed by Eric Yang 2018-12-14 18:44:36 -08:00
Eric Yang
b2d7204ed0 YARN-9125. Fixed Carriage Return detection in Docker container launch command. 
Contributed by Billie Rinaldi
 2018-12-14 17:52:26 -05:00
Billie Rinaldi
ca90b2480e YARN-9032. Support sh shell for interactive container shell at command line. Contributed by Eric Yang 2018-12-14 14:08:22 -08:00
Billie Rinaldi
72e7c6a489 YARN-8962. Add ability to use interactive shell with normal yarn container. Contributed by Eric Yang 2018-12-12 18:18:57 -08:00
Wangda Tan
37eb919c59 YARN-8885. [DevicePlugin] Support NM APIs to query device resource allocation. (Zhankun Tang via wangda) 
Change-Id: I2a9870709b512af1ac6c09c9701d0b3c0791ff32
 2018-12-12 11:45:47 -08:00
Wangda Tan
61bdcb7b2b YARN-9015. [DevicePlugin] Add an interface for device plugin to provide customized scheduler. (Zhankun Tang via wangda) 
Change-Id: Ib2e4ae47a6f29bb3082c1f8520cf5a52ca720979
 2018-12-12 11:44:22 -08:00
Haibo Chen
881230da21 YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth) 2018-12-11 11:41:32 -08:00
Haibo Chen
ac578c0e82 YARN-9087. Improve logging for initialization of Resource plugins. (Contributed by Szilard Nemeth) 2018-12-10 11:12:54 -08:00
Billie Rinaldi
154449fbd8 YARN-8914. Add xtermjs to YARN UI2. Contributed by Eric Yang and Akhil PB 2018-12-07 10:56:17 -08:00
Eric Yang
1b790f4dd1 YARN-9071. Improved status update for reinitialized containers. 
Contributed by Chandni Singh
 2018-12-05 17:00:56 -05:00
Wangda Tan
bad12031f6 YARN-9010. Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl. (Zhankun Tang via wangda) 
Change-Id: Iaecc66d57781cc10f19ead4647e47fc9556676da
 2018-11-29 14:56:07 -08:00
Wangda Tan
fe7dab8ef5 YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun Tang via wangda) 
Change-Id: Ia6d532c687168448416dfdf46f0ac34bff20e6ca
 2018-11-28 15:03:06 -08:00
Wangda Tan
8ebeda98a9 YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun Tang via wangda) 
Change-Id: I4eb58e9d251d5f54e7feffc4fbb813b4f5ae4b1b
 2018-11-28 14:36:30 -08:00
Wangda Tan
9ed87567ad YARN-9061. Improve the GPU/FPGA module log message of container-executor. (Zhankun Tang via wangda) 
Change-Id: Iece9b47438357077a53984a820d4d6423f480518
 2018-11-28 14:31:31 -08:00
Wangda Tan
579ef4be06 YARN-8882. [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins. (Zhankun Tang via wangda) 
Change-Id: I9435136642c3d556971a357bf687f69df90bb45e
 2018-11-28 14:09:52 -08:00
Jason Lowe
3ce99e32f7 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko 2018-11-28 08:46:11 -06:00
Eric Yang
33e0df4b35 YARN-8986. Added port publish for Docker container running with bridge network. 
Contributed by Charo Zhang
 2018-11-26 19:45:05 -05:00
Rohith Sharma K S
e223a790a7 YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun Tang. 2018-11-22 20:04:30 +05:30
Billie Rinaldi
49824ed260 YARN-8838. Check that container user is same as websocket user for interactive shell. Contributed by Eric Yang 2018-11-20 11:12:24 -08:00
Wangda Tan
6357803645 YARN-8881. [YARN-8851] Add basic pluggable device plugin framework. (Zhankun Tang via wangda) 
Change-Id: If9a2f68cd4713b4ec932cdeda68106f17437c3d3
 2018-11-19 08:54:31 -08:00
Eric Yang
21ec4bdaef YARN-8672. Improve token filename management for localization. 
Contributed by Chandni Singh
 2018-11-14 15:22:01 -05:00
Billie Rinaldi
1f9c4f32e8 YARN-8776. Implement Container Exec feature in LinuxContainerExecutor. Contributed by Eric Yang 2018-11-12 10:42:30 -08:00
Botong Huang
b5ec85d966 YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang. 2018-11-11 11:12:53 -08:00
Weiwei Yang
f8c72d7b3a YARN-8880. Add configurations for pluggable plugin framework. Contributed by Zhankun Tang. 2018-11-08 12:23:00 +08:00
Giovanni Matteo Fumarola
989715ec50 YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang. 2018-11-02 15:30:08 -07:00
Billie Rinaldi
d07e873b7d YARN-8569. Create an interface to provide cluster information to application. Contributed by Eric Yang 2018-10-26 17:57:05 -07:00
Robert Kanter
f76e3c3db7 YARN-8930. CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController (haibochen via rkanter) 2018-10-25 11:09:47 -07:00
Eric Yang
c16c49b8c3 YARN-8587. Added retries for fetching docker exit code. 
Contributed by Charo Zhang
 2018-10-24 17:29:46 -04:00
Robert Kanter
69b328943e YARN-8929. DefaultOOMHandler should only pick running containers to kill upon oom events (haibochen via rkanter) 2018-10-24 13:15:50 -07:00
Haibo Chen
766b78ee07 YARN-8911. ContainerScheduler incorrectly uses percentage number as the cpu resource utlization. 2018-10-24 07:58:26 -07:00
Eric Yang
66e7a2c787 YARN-8922. Fixed test-container-executor test setup and clean up. 
Contributed by Robert Kanter
 2018-10-22 15:28:58 -04:00
Eric Yang
e2a9fa8448 HADOOP-15821. Move YARN Registry to Hadoop Registry. 
Contributed by Íñigo Goiri
 2018-10-19 19:46:48 -04:00
Eric Yang
47ad98b2e1 YARN-8910. Fixed misleading log statement when container max retries is infinite. 
Contributed by Chandni Singh
 2018-10-19 13:49:04 -04:00
Wangda Tan
5e02b4915b YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda) 
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
 2018-10-19 09:49:26 -07:00
Wangda Tan
a457a8951a YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda) 
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
 2018-10-18 10:57:11 -07:00
Haibo Chen
32fe351bb6 YARN-8864. NM incorrectly logs container user as the user who sent a start/stop container request in its audit log. (Contributed by Wilfred Spiegelenburg) 2018-10-18 08:28:07 -07:00
Haibo Chen
c2288ac45b YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter) 2018-10-16 13:36:26 -07:00
Haibo Chen
f880ff418c YARN-8775. TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications. (Contributed by Antal Bálint Steinbach) 2018-10-15 09:37:20 -07:00
Billie Rinaldi
96d28b4750 YARN-8777. Container Executor C binary change to execute interactive docker command. Contributed by Eric Yang 2018-10-11 09:25:21 -07:00
Jason Lowe
e787d65a08 YARN-8861. executorLock is misleading in ContainerLaunch. Contributed by Chandni Singh 2018-10-11 10:54:57 -05:00
Jason Lowe
5ce70e1211 YARN-7644. NM gets backed up deleting docker containers. Contributed by Chandni Singh 2018-10-10 09:52:19 -05:00
Inigo Goiri
c3d22d3b45 YARN-7652. Handle AM register requests asynchronously in FederationInterceptor. Contributed by Botong Huang. 2018-10-09 10:29:40 -07:00
Eric Yang
347ea38581 YARN-8763. Added node manager websocket API for accessing containers. 
Contributed by Zian Chen
 2018-10-08 13:56:47 -04:00
Wangda Tan
2e9913caf2 YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda) 
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
(cherry picked from commit b3ac8869338faeabedbc27e95e3166c830a9a761)
 2018-10-04 10:53:39 -07:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Shane Kumpf
5edb9d3b97 YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo 2018-10-02 07:16:29 -06:00
Giovanni Matteo Fumarola
59d5af21b7 YARN-8760. [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer. Contributed by Botong Huang. 2018-10-01 13:12:38 -07:00
Eric Yang
b237a0dd44 YARN-6456. Added config to set default container runtimes. 
Contributed by Craig Condit
 2018-09-27 15:31:18 -04:00
Eric Yang
913f87dada YARN-8665. Added Yarn service cancel upgrade option. 
Contributed by Chandni Singh
 2018-09-26 14:51:35 -04:00
Giovanni Matteo Fumarola
3090922805 YARN-8696. [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async. Contributed by Botong Huang. 2018-09-24 11:37:05 -07:00
Eric Yang
aa4bd493c3 YARN-8801. Fixed header comments for docker utility functions. 
Contributed by Zian Chen
 2018-09-20 13:08:59 -04:00
Jason Lowe
6b5838ed32 YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger 2018-09-19 16:44:51 -05:00
Eric Yang
efdea85ad1 YARN-8791. Trim docker inspect output for line feed for STOPSIGNAL parsing. 
Contributed by Chandni Singh
 2018-09-19 13:16:11 -04:00
Jason Lowe
2df0a8dcb3 YARN-8648. Container cgroups are leaked when using docker. Contributed by Jim Brennan 2018-09-18 15:36:45 -05:00
Shane Kumpf
144a55f0e3 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 2018-09-14 10:41:55 -06:00
Shane Kumpf
78902f0250 YARN-8748. Javadoc warnings within the nodemanager package. Contributed by Craig Condit 2018-09-14 10:28:36 -06:00
Eric Yang
99237607bf YARN-8706. Allow additional flag in docker inspect call. 
Contributed by Chandni Singh
 2018-09-14 11:46:59 -04:00
Jason Lowe
250b50018e YARN-8680. YARN NM: Implement Iterable Abstraction for LocalResourceTracker state. Contributed by Pradeep Ambati 2018-09-13 13:28:54 -05:00
Weiwei Yang
39c1ea1ed4 YARN-8729. Node status updater thread could be lost after it is restarted. Contributed by Tao Yang. 2018-09-13 22:21:35 +08:00
Giovanni Matteo Fumarola
02b9bfdf9e YARN-8658. [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor. Contributed by Young Chen. 2018-09-12 11:46:35 -07:00
bibinchundatt
c44088ac19 YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
bibinchundatt
8cf6a9a2bd YARN-7892. Revisit NodeAttribute class structure. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Sunil G
440ff7f563 YARN-8094. Support configuration based Node Attribute provider. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Naganarasimha
3b3b6efe21 YARN-7871. Node attributes reporting from NM to RM. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G
ffcabd24c3 YARN-7856. Validate Node Attributes from NM. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G
2f7712be09 YARN-6858. Attribute Manager to store and provide node attributes in RM. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30