big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,591 Commits 2 Branches 2 Tags 264 MiB
d7979079ea
Commit Graph

5199 Commits

Author SHA1 Message Date
Eric Yang
01ada40ea4 YARN-9348. Application catalog build system bug fixes. 
Contributed by Eric Yang
 2019-03-06 17:55:48 -05:00
Sunil G
c79f139519 YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB. 2019-03-06 16:49:08 +05:30
Sunil G
46045c5cb3 YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-06 16:01:08 +05:30
Eric Yang
7b42e0e32a YARN-7266. Fixed deadlock in Timeline Server thread initialization. 
Contributed by Prabhu Joseph
 2019-03-05 12:17:01 -05:00
Billie Rinaldi
2576401928 YARN-7129. Application Catalog for YARN applications. Contributed by Eric Yang 2019-03-05 06:14:14 -08:00
Yufei Gu
0aefe2846f YARN-9298. Implement FS placement rules using PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-04 23:49:07 -08:00
Prabhu Joseph
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 14:10:08 +09:00
bibinchundatt
15098df744 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph." 
This reverts commit a63c358b78.
 2019-03-04 16:57:31 +05:30
Akira Ajisaka
bd8d299ded
YARN-7477. Moving logging APIs over to slf4j in hadoop-yarn-common. Contributed by Prabhu Joseph. 2019-03-04 16:59:20 +09:00
Weiwei Yang
e20b5ef52c
YARN-9332. RackResolver tool should accept multiple hosts. Contributed by Lantao Jin. 2019-03-04 16:59:20 +09:00
Suma Shivaprasad
cab8529ecb YARN-7904. Privileged, trusted containers should be supported only in ENTRYPOINT mode. Contributed by Eric Yang. 2019-03-01 11:06:09 -08:00
Sunil G
dcaca19871 YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-01 19:24:35 +05:30
Szilard Nemeth
538bb4880d YARN-9323. FSLeafQueue#computeMaxAMResource does not override zero values for custom resources 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: Id844ccf09488f367c0c7de0a3b2d4aca1bba31cc
 2019-02-27 19:59:48 -08:00
Szilard Nemeth
7b928f19a4 YARN-9322. Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: I14c12f1265999d62102f2ec5506d90015efeefe8
 2019-02-27 19:43:50 -08:00
Weiwei Yang
1779fc57a1 YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph. 2019-02-28 09:56:29 +08:00
Eric Yang
04b228e43b YARN-9334. Allow YARN Service client to send SPNEGO challenge header when authentication type is not simple. 
Contributed by Billie Rinaldi
 2019-02-27 18:47:14 -05:00
Vrushali C
ea3cdc60b3 YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS backing storage. Contributed by Abhishek Modi. 2019-02-27 14:55:35 -08:00
Vrushali C
0ec962ac8f YARN-5336 Limit the flow name size & consider cleanup for hex chars. Contributed by Sushil Ks 2019-02-27 14:43:39 -08:00
Eric Yang
fbc7bb315f YARN-9245. Added query docker image command ability to node manager. 
Contributed by Chandni Singh
 2019-02-27 14:57:24 -05:00
Szilard Nemeth
29e27faf96 YARN-9318. Resources#multiplyAndRoundUp does not consider Resource Types 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: Ia45f528574c2b054f6f764d1d140e592bdb7e217
 2019-02-27 10:29:17 -08:00
Weiwei Yang
8c30114b00 YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie. 2019-02-27 17:29:02 +08:00
Rohith Sharma K S
6c96f5e4b6 YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor filters. Contributed by Lantao Jin. 2019-02-27 10:32:58 +05:30
Rohith Sharma K S
8eae260af5 YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph. 2019-02-27 10:28:16 +05:30
Suma Shivaprasad
a106d2dc9d YARN-8783. Improve the documentation for the docker.trusted.registries configuration. Contributed by Eric Yang. 2019-02-26 10:19:06 -08:00
Weiwei Yang
c6ea28c480 YARN-9331. [YARN-8851] Fix a bug that lacking cgroup initialization when bootstrap DeviceResourceHandlerImpl. Contributed by Zhankun Tang. 2019-02-26 10:05:31 +08:00
Giovanni Matteo Fumarola
95372657fc YARN-9287. Consecutive StringBuilder append should be reuse. Contributed by Ayush Saxena. 2019-02-25 11:45:37 -08:00
Weiwei Yang
3e1739d589 YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang. 2019-02-26 00:08:13 +08:00
Sunil G
6cec90653d YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang. 2019-02-25 15:28:45 +05:30
Sunil G
5e91ebd91a YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth. 2019-02-25 11:30:46 +05:30
Weiwei Yang
9cd5c5447f YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph. 2019-02-24 22:42:27 +08:00
Weiwei Yang
50094d7fef YARN-9300. Lazy preemption should trigger an update on queue preemption metrics for CapacityScheduler. Contributed by Tao Yang. 2019-02-24 22:17:29 +08:00
Sunil G
dddcfa4d9f YARN-8821. [YARN-8851] GPU hierarchy/topology scheduling support based on pluggable device framework. Contributed by Zhankun Tang. 2019-02-24 14:37:06 +05:30
Eric Yang
05bce33d6e YARN-9244. Document docker registry deployment with direct S3 driver. 
Contributed by Suma Shivaprasad
 2019-02-22 19:13:52 -05:00
bibinchundatt
ed13cf8406 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-22 21:01:34 +05:30
Sunil G
95fbbfed75 YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth. 2019-02-22 20:22:17 +05:30
Sunil G
9636fe4114 YARN-8891. Documentation of the pluggable device framework. Contributed by Zhankun Tang. 2019-02-22 20:00:13 +05:30
Weiwei Yang
9c88695bcd YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie. 2019-02-22 20:58:33 +08:00
Weiwei Yang
632d5e8a98 YARN-9156. Improve debug message in device plugin method compatibility check of ResourcePluginManager. Contributed by Zhankun Tang. 2019-02-22 11:27:30 +08:00
Wangda Tan
b17a2602d1 YARN-9319. Fix compilation issue of handling typedef an existing name by gcc compiler. (Zhankun Tang via wangda) 
Change-Id: Ie8293cc5b710ff355fb6a1ea7201be53ed0385bf
 2019-02-21 11:17:32 -08:00
Weiwei Yang
6c4ab0312b YARN-9258. Support to specify allocation tags without constraint in distributed shell CLI. Contributed by Prabhu Joseph. 2019-02-22 00:18:07 +08:00
Weiwei Yang
eedcc8e26e YARN-9315. TestCapacitySchedulerMetrics fails intermittently. Contributed by Prabhu Joseph. 2019-02-21 17:21:21 +08:00
bibinchundatt
a63c358b78 YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph. 2019-02-21 13:29:01 +05:30
bibinchundatt
b8de78c570 YARN-9286. [Timeline Server] Sorting based on FinalStatus shows pop-up message. Contributed by Bilwa S T. 2019-02-20 01:19:20 +05:30
Sunil G
db4d1a1e2f YARN-9060. [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example. Contributed by Zhankun Tang. 2019-02-18 15:58:04 +05:30
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Sunil G
ba56bc24f5 YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth. 2019-02-17 20:08:20 +05:30
Sunil G
9057aa9fc5 SUBMARINE-1. Move code base of submarine from yarn-applications to top directory. Contributed by Wangda Tan. 2019-02-17 20:03:56 +05:30
Eric Yang
7c1b561e33 YARN-8927. Added support for top level Dockerhub images to trusted registry using library keyword. 
Contributed by Zhankun Tang
 2019-02-15 11:18:07 -05:00
Adam Antal
9385ec45d7
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-02-15 18:47:10 +09:00
Sunil G
5b55f3538c YARN-8295. [UI2] Improve Resource Usage tab error message when there are no data available. Contributed by Charan Hebri. 2019-02-15 12:42:21 +05:30
Akira Ajisaka
3a39d9a2d2
YARN-9284. Fix the unit of yarn.service.am-resource.memory in the document. Contributed by Masahiro Tanaka. 2019-02-15 15:41:53 +09:00
bibinchundatt
b66d5ae9e2 YARN-9295. [UI2] Fix label typo in Cluster Overview page. Contributed by Charan Hebri. 2019-02-14 23:08:19 +05:30
Sunil G
dfe0f42835 YARN-7824. [UI2] Yarn Component Instance page should include link to container logs. Contributed by Akhil PB. 2019-02-14 15:42:19 +05:30
Eric Yang
3dc2523266 YARN-9184. Add a system flag to allow update to latest docker images. 
Contributed by Zhaohui Xin
 2019-02-12 16:16:35 -05:00
Giovanni Matteo Fumarola
7536488bbd YARN-996. REST API support for node resource configuration. Contributed by Inigo Goiri. 2019-02-11 15:27:43 -08:00
Suma Shivaprasad
1ce2e91c4b YARN-9229. Document docker registry deployment with NFS Gateway. Contributed by Eric Yang. 2019-02-11 14:54:30 -08:00
Weiwei Yang
0a1637c750 YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options. Contributed by Prabhu Joseph. 2019-02-11 15:53:50 +08:00
Weiwei Yang
2b7f828d46 YARN-9252. Allocation Tag Namespace support in Distributed Shell. Contributed by Prabhu Joseph. 2019-02-11 14:42:21 +08:00
Masatake Iwasaki
e0ab1bdece YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 2019-02-09 00:26:50 +09:00
Eric E Payne
d1ca9432dd YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 2019-02-07 16:38:11 +00:00
Vinayakumar B
3c96a03dea YARN-8498. Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04. Contributed by Ayush Saxena. 2019-02-07 13:02:35 +05:30
Rohith Sharma K S
7fa62e150c YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi. 2019-02-07 09:58:53 +05:30
Rohith Sharma K S
1e5e08d83b YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error message is wrong. Contributed by Charan Hebri. 2019-02-07 09:58:53 +05:30
Weiwei Yang
711d22f166 YARN-9253. Add UT to verify Placement Constraint in Distributed Shell. Contributed by Prabhu Joseph. 2019-02-06 22:36:51 +08:00
Wangda Tan
308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Suma Shivaprasad
2044967e75 YARN-9246 NPE when executing a command yarn node -status or -states without additional arguments. Contributed by Masahiro Tanaka 2019-02-05 09:39:38 -08:00
Rohith Sharma K S
e3ec18b0c4 YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi. 2019-02-05 13:48:04 +05:30
Sunil G
ba38db4f5b YARN-9257. Distributed Shell client throws a NPE for a non-existent queue. Contributed by Charan Hebri. 2019-02-05 13:41:44 +05:30
Weiwei Yang
aa7ce50e55 YARN-9275. Add link to NodeAttributes doc in PlacementConstraints document. Contributed by Masatake Iwasaki. 2019-02-05 16:07:41 +08:00
Rohith Sharma K S
571838920d YARN-9149. yarn container -status misses logUrl when integrated with ATSv2. Contributed by Abhishek Modi. 2019-02-05 12:39:10 +05:30
Sunil G
604b2489a9 YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 2019-02-04 12:48:28 +05:30
Weiwei Yang
69bcff380f YARN-9231. TestDistributedShell fix timeout. Contributed by Prabhu Joseph. 2019-02-02 23:27:31 +08:00
Weiwei Yang
f20b043a02 YARN-9263. TestConfigurationNodeAttributesProvider fails after Mockito updated. Contributed by Weiwei Yang. 2019-02-02 23:04:34 +08:00
Weiwei Yang
28ad20a711 YARN-9262. TestRMAppAttemptTransitions is failing with an NPE. Contributed by lujie. 2019-02-02 23:02:45 +08:00
Vrushali C
2c13513040 YARN-8549 Adding a NoOp timeline writer and reader plugin classes for ATSv2. Contributed by Prabha Manepalli. 2019-02-01 14:26:50 -08:00
Giovanni Matteo Fumarola
f738b397ae YARN-9191. Add cli option in DS to support enforceExecutionType in resource requests. Contributed by Abhishek Modi. 2019-01-31 11:24:15 -08:00
Sunil G
71c49fa60f YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way. Contributed by Szilard Nemeth. 2019-01-31 09:25:29 +05:30
Akira Ajisaka
a3a9ae3cea
YARN-9251. Build failure for -Dhbase.profile=2.0. Contributed by Rohith Sharma K S. 2019-01-30 05:17:04 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Eric E Payne
04105bbfdb YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 2019-01-29 17:22:24 +00:00
Weiwei Yang
085f0e8ae7 YARN-9086. [CSI] Run csi-driver-adaptor as aux service. Contributed by Weiwei Yang. 2019-01-29 14:53:08 +08:00
Weiwei Yang
4f63ffe444 YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang. 2019-01-29 10:34:26 +08:00
Eric Yang
f5a95f7998 YARN-8901. Fixed restart policy NEVER/ON_FAILURE with component dependency. 
Contributed by Suma Shivaprasad
 2019-01-28 18:10:33 -05:00
Eric Yang
2e636dd3c4 YARN-9074. Consolidate docker removal logic in ContainerCleanup. 
Contributed by Zhaohui Xin
 2019-01-28 18:05:53 -05:00
Eric Yang
1ab69a9543 YARN-9221. Added flag to disable dynamic auxiliary service feature. 
Contributed by Billie Rinaldi
 2019-01-25 19:05:36 -05:00
Jonathan Hung
6cace58e21 YARN-9222. Print launchTime in ApplicationSummary 2019-01-25 13:17:00 -08:00
Rohith Sharma K S
84bb9808a2 YARN-7761. [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link. Contributed by Akhil PB. 2019-01-25 14:00:35 +05:30
Eric Yang
a33ef4fd31 YARN-8867. Added resource localization status to YARN service status call. 
Contributed by Chandni Singh
 2019-01-24 18:43:21 -05:00
Weiwei Yang
a4bd64e724 YARN-9116. Capacity Scheduler: implements queue level maximum-allocation inheritance. Contributed by Aihua Xu. 2019-01-24 17:53:43 +08:00
Sunil G
c726445990 YARN-8961. [UI2] Flow Run End Time shows 'Invalid date'. Contributed by Akhil PB 2019-01-24 15:01:43 +05:30
Weiwei Yang
721d5c2a5f YARN-8101. Add UT to verify node-attributes in RM nodes rest API. Contributed by Prabhu Joseph. 2019-01-23 17:59:36 +08:00
Weiwei Yang
e3e076dc78 YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 2019-01-23 14:14:49 +08:00
Eric Yang
2fa9389c2e YARN-9146. Added REST API to configure auxiliary service. 
Contributed by Billie Rinaldi
 2019-01-22 18:24:43 -05:00
Weiwei Yang
d43df31751 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 2019-01-22 10:23:32 +08:00
Weiwei Yang
abde1e1f58 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang. 2019-01-21 16:54:14 +08:00
Takanobu Asanuma
27aa6e8899 HADOOP-16046. [JDK 11] Correct the compiler exclusion of org/apache/hadoop/yarn/webapp/hamlet/** classes for >= Java 9. Contributed by Devaraj K. 2019-01-21 10:54:58 +09:00
Eric Yang
824dfa3b09 YARN-8489. Support "dominant" component concept in YARN service. 
Contributed by Zac Zhou
 2019-01-18 20:23:04 -05:00
Eric Yang
8c7f6b2d4d YARN-9197. Add safe guard against NPE for component instance failure. 
Contributed by kyungwan nam
 2019-01-18 13:49:56 -05:00
Wangda Tan
6d7eedfd28 YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda) 
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
 2019-01-17 14:20:10 -08:00
rahul3
0a46baecd3
YARN-9203. Fix typos in yarn-default.xml. 
This closes #437

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-01-17 16:18:33 +09:00