big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,662 Commits 2 Branches 2 Tags 264 MiB
69ddb36876
Commit Graph

3855 Commits

Author SHA1 Message Date
Eric Yang
29465bf169 YARN-9560. Restructure DockerLinuxContainerRuntime to extend OCIContainerRuntime. 
Contributed by Eric Badger, Jim Brennan, Craig Condit
 2019-06-28 17:18:53 -04:00
Weiwei Yang
f09c31a97e Revert "YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023" 
This reverts commit 5e7caf1287.
 2019-06-29 00:29:17 +08:00
Weiwei Yang
5e7caf1287 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023 2019-06-29 00:08:40 +08:00
Weiwei Yang
cbae241320 YARN-9623. Auto adjust max queue length of app activities to make sure activities on all nodes can be covered. Contributed by Tao Yang. 2019-06-28 23:24:53 +08:00
bibinchundatt
be80334cdf YARN-9639. DecommissioningNodesWatcher cause memory leak. Contributed by Bilwa S T. 2019-06-27 09:59:44 +05:30
Giovanni Matteo Fumarola
1ac967a6b7 YARN-6055. ContainersMonitorImpl need be adjusted when NM resource changed. Contributed by Inigo Goiri. 2019-06-26 14:01:31 -07:00
Zhankun Tang
062eb605ac YARN-9477. Implement VE discovery using libudev. Contributed by Peter Bacsko. 2019-06-26 23:53:14 +08:00
Eric Yang
b220ec6f61 YARN-9374. Improve Timeline service resilience when HBase is unavailable. 
Contributed by Prabhu Joseph and Szilard Nemeth
 2019-06-24 12:19:14 -04:00
Weiwei Yang
83dcb9d87e YARN-9209. When nodePartition is not set in Placement Constraints, containers are allocated only in default partition. Contributed by Tarun Parimi. 2019-06-21 17:41:05 +08:00
Zhankun Tang
67414a1a80 YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 2019-06-18 12:23:52 +08:00
Zhankun Tang
304a47e22c YARN-9608. DecommissioningNodesWatcher should get lists of running applications on node from RMNode. Contributed by Abhishek Modi. 2019-06-17 17:09:56 +08:00
Eric Yang
cda9f33745 YARN-8499 ATSv2 Generalize TimelineStorageMonitor. 
Contributed by Prabhu Joseph
 2019-06-14 18:59:14 -04:00
Eric Yang
3ba090f436 HADOOP-16366. Fixed ProxyUserAuthenticationFilterInitializer for timeline server. 
Contributed by Prabhu Joseph
 2019-06-14 12:54:16 -04:00
Giovanni Matteo Fumarola
bcfd228336 YARN-9599. TestContainerSchedulerQueuing#testQueueShedding fails intermittently. Contributed by Abhishek Modi. 2019-06-13 11:08:35 -07:00
Weiwei Yang
970b0b0c02 YARN-9578. Add limit/actions/summarize options for app activities REST API. Contributed by Tao Yang. 2019-06-13 10:44:47 +08:00
Eric Yang
205dd2d8e1 HADOOP-16367. Fixed MiniYarnCluster AuthenticationFilter initialization. 
Contributed by Prabhu Joseph
 2019-06-12 18:03:33 -04:00
bibinchundatt
2263ead365 YARN-9557. Application fails in diskchecker when ReadWriteDiskValidator is configured. Contributed by Bilwa S T. 2019-06-11 23:20:28 +05:30
bibinchundatt
60c95e9b6a YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T. 2019-06-11 23:11:49 +05:30
bibinchundatt
6d80b9bc3f YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 2019-06-11 22:49:21 +05:30
bibinchundatt
f7df55f4a8 YARN-9602. Use logger format in Container Executor. Contributed by Abhishek Modi. 2019-06-11 22:29:00 +05:30
Suma Shivaprasad
9191e08f0a YARN-9569. Auto-created leaf queues do not honor cluster-wide min/max memory/vcores. Contributed by Craig Condit. 2019-06-10 14:33:24 -07:00
Weiwei Yang
0976392502 YARN-9590. Correct incompatible, incomplete and redundant activities. Contributed by Tao Yang. 2019-06-06 21:59:01 +08:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Weiwei Yang
433e97cd34 YARN-9600. Support self-adaption width for columns of containers table on app attempt page. Contributed by Tao Yang. 2019-06-05 13:55:30 +08:00
Eric Yang
d45669cd3c YARN-7537. Add ability to load hbase config from distributed file system. 
Contributed by Prabhu Joseph
 2019-06-04 19:26:06 -04:00
Zhankun Tang
606061aa14 YARN-9595. FPGA plugin: NullPointerException in FpgaNodeResourceUpdateHandler.updateConfiguredResource(). Contributed by Peter Bacsko. 2019-06-04 09:56:59 +08:00
Weiwei Yang
bd2590d71b YARN-9580. Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers. Contributed by Tao Yang. 2019-06-03 22:59:02 +08:00
Weiwei Yang
4530f4500d YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T. 2019-06-03 14:09:37 +08:00
Giovanni Matteo Fumarola
2210897609 YARN-9592. Use Logger format in ContainersMonitorImpl. Contributed by Inigo Goiri. 2019-05-31 17:35:49 -07:00
Eric Yang
4cb559ea7b YARN-9027. Fixed LevelDBCacheTimelineStore initialization. 
Contributed by Prabhu Joseph
 2019-05-31 14:31:44 -04:00
Sunil G
e49162f4b3 YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 2019-05-31 10:28:09 +05:30
Sunil G
7861a5eb1a YARN-9033. ResourceHandlerChain#bootstrap is invoked twice during NM start if LinuxContainerExecutor enabled. Contributed by Zhankun Tang. 2019-05-31 10:22:26 +05:30
Giovanni Matteo Fumarola
f1552f6edb YARN-9553. Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines. Contributed by Prabhu Joseph. 2019-05-30 11:42:27 -07:00
Sunil G
30c6dd92e1 YARN-9452. Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures. Contributed by Prabhu Joseph. 2019-05-30 22:32:41 +05:30
Ahmed Hussein
abf76ac371 YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-05-29 11:24:08 -05:00
Eric E Payne
3c63551101 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 2019-05-29 16:05:39 +00:00
Weiwei Yang
544876fe12 YARN-8693. Add signalToContainer REST API for RMWebServices. Contributed by Tao Yang. 2019-05-29 16:34:48 +08:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9078e28a24
YARN-9503. Fix JavaDoc error in TestSchedulerOvercommit. Contributed by Wanqiang Ji. 2019-05-28 15:52:39 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Weiwei Yang
9f056d905f YARN-9497. Support grouping by diagnostics for query results of scheduler and app activities. Contributed by Tao Yang. 2019-05-26 09:56:36 -04:00
Eric Yang
460ba7fb14 YARN-9558. Fixed LogAggregation test cases. 
Contributed by Prabhu Joseph
 2019-05-23 18:38:47 -04:00
Eric Yang
7b03072fd4 YARN-9080. Added clean up of bucket directories. 
Contributed by Prabhu Joseph, Peter Bacsko, Szilard Nemeth
 2019-05-23 12:08:44 -04:00
Giovanni Matteo Fumarola
12c81610e0 YARN-9505. Add container allocation latency for Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-05-17 12:03:21 -07:00
Eric Yang
fab5b80a36 YARN-9554. Fixed TimelineEntity DAO serialization handling. 
Contributed by Prabhu Joseph
 2019-05-16 16:39:50 -04:00
Giovanni Matteo Fumarola
55bd35921c YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-05-15 11:50:46 -07:00
bibinchundatt
570fa2da20 YARN-9508. YarnConfiguration areNodeLabel enabled is costly in allocation flow. Contributed by Bilwa S T. 2019-05-15 13:30:09 +05:30
bibinchundatt
2de1e30658 YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T. 2019-05-15 13:21:39 +05:30
Giovanni Matteo Fumarola
29ff7fb140 YARN-9493. Scheduler Page does not display the right page by query string. Contributed by Wanqiang Ji. 2019-05-13 10:57:12 -07:00
Weiwei Yang
1a47c2b7ae YARN-9539.Improve cleanup process of app activities and make some conditions configurable. Contributed by Tao Yang. 2019-05-12 22:31:39 -07:00
Giovanni Matteo Fumarola
1b48100a5e YARN-9522. AppBlock ignores full qualified class name of PseudoAuthenticationHandler. Contributed by Prabhu Joseph. 2019-05-09 14:02:58 -07:00
Weiwei Yang
90add05caa YARN-9489. Support filtering by request-priorities and allocation-request-ids for query results of app activities. Contributed by Tao Yang. 2019-05-09 21:54:09 +08:00
Akira Ajisaka
3172f6cbf9
YARN-9513. Addendum patch: Fix ASF License warnings. Contributed by Giovanni Matteo Fumarola. 2019-05-08 14:56:23 +09:00
Weiwei Yang
c336af3847 YARN-9432. Reserved containers leak after its request has been cancelled or satisfied when multi-nodes enabled. Contributed by Tao Yang. 2019-05-08 09:54:16 +08:00
Giovanni Matteo Fumarola
8ecbf61cca YARN-9513. [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8. Contributed by Adam Antal. 2019-05-07 10:59:02 -07:00
Haibo Chen
597fa47ad1 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 2019-05-06 11:56:22 -07:00
Weiwei Yang
12b7059ddc YARN-9440. Improve diagnostics for scheduler and app activities. Contributed by Tao Yang. 2019-05-06 20:00:15 +08:00
Eric E Payne
b094b94d43 YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 2019-05-02 19:39:26 +00:00
Eric Yang
accb811e57 YARN-6929. Improved partition algorithm for yarn remote-app-log-dir. 
Contributed by Prabhu Joseph
 2019-04-30 17:04:59 -04:00
Zhankun Tang
7fbaa7d66f YARN-9476. [YARN-9473] Create unit tests for VE plugin. Contributed by Peter Bacsko. 2019-04-30 11:06:44 +08:00
Eric Badger
79d3d35398 YARN-9486. Docker container exited with failure does not get clean up correctly. Contributed by Eric Yang 2019-04-26 01:21:28 +00:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Giovanni Matteo Fumarola
3f2f4186f6 YARN-9424. Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent. Contributed by Shen Yinjie. 2019-04-23 19:58:41 -07:00
Giovanni Matteo Fumarola
fec9bf4b0b YARN-9501. TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 15:42:56 -07:00
Giovanni Matteo Fumarola
4a0ba24959 YARN-9491. TestApplicationMasterServiceFair#ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 15:27:04 -07:00
Inigo Goiri
c504eee0c2 YARN-9339. Apps pending metric incorrect after moving app to a new queue. Contributed by Abhishek Modi. 2019-04-23 12:40:44 -07:00
Zhankun Tang
8a95ea61e1 YARN-9475. [YARN-9473] Create basic VE plugin. Contributed by Peter Bacsko. 2019-04-23 17:33:58 +08:00
Weiwei Yang
1c8046d67e YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 14:21:13 +08:00
Inigo Goiri
96e3027e46 YARN-2889. Limit the number of opportunistic container allocated per AM heartbeat. Contributed by Abhishek Modi. 2019-04-22 09:49:03 -07:00
Inigo Goiri
aeadb9432f YARN-9448. Fix Opportunistic Scheduling for node local allocations. Contributed by Abhishek Modi. 2019-04-19 09:41:06 -07:00
Eric Yang
ef97a20831 YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng
 2019-04-18 18:59:21 -04:00
Eric Yang
df76cdc895 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph
 2019-04-18 12:29:37 -04:00
Prabhu Joseph
aa4c744aef
YARN-9470. Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-18 15:40:37 +09:00
Siyao Meng
6e4399ea61 YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-17 22:56:57 -07:00
Eric Yang
9cf7401794 YARN-9349. Improved log level practices for InvalidStateTransitionException. 
Contributed by Anuhan Torgonshar

(cherry picked from commit fe2370e039e1ee980d74769ae85d67434e0993cf)
 2019-04-16 19:53:45 -04:00
Szilard Nemeth
b8086aed86 YARN-9123. Clean up and split testcases in TestNMWebServices for GPU support. Contributed by Szilard Nemeth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-16 11:06:25 -07:00
Eric Badger
5583e1b6fc YARN-7848 Force removal of docker containers that do not get removed on first try. Contributed by Eric Yang 2019-04-15 20:47:09 +00:00
Eric Badger
254efc9358 YARN-9379. Can't specify docker runtime through environment. Contributed by caozhiqiang 2019-04-15 18:24:37 +00:00
Weiwei Yang
7fa73fac26 YARN-9439. Support asynchronized scheduling mode and multi-node lookup mechanism for app activities. Contributed by Tao Yang. 2019-04-16 00:12:43 +08:00
Inigo Goiri
7a68e7abd5 YARN-9474. Remove hard coded sleep from Opportunistic Scheduler tests. Contributed by Abhishek Modi. 2019-04-14 20:11:20 -07:00
Gabor Bota
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre. 
Author:    Gabor Bota <gabor.bota@cloudera.com>
 2019-04-12 18:28:38 -07:00
Giovanni Matteo Fumarola
ed3747c1cc YARN-9435. Add Opportunistic Scheduler metrics in ResourceManager. Contributed by Abhishek Modi. 2019-04-11 11:49:19 -07:00
Weiwei Yang
8c1bba375b YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 2019-04-10 22:51:28 +08:00
Igor Rudenko
32722d2661
YARN-9433. Remove unused constants in YARN resource manager 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-10 18:37:27 +09:00
Giovanni Matteo Fumarola
cfec455c45 YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Contributed by Inigo Goiri. 2019-04-09 10:59:43 -07:00
Weiwei Yang
fc05b0e70e YARN-9313. Support asynchronized scheduling mode and multi-node lookup mechanism for scheduler activities. Contributed by Tao Yang. 2019-04-08 13:40:53 +08:00
Weiwei Yang
ec143cbf67 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 2019-04-06 19:59:36 +08:00
Vrushali C
22362c876d YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi. 2019-04-05 12:06:51 -07:00
Vrushali C
27039a29ae YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. 2019-04-05 12:02:43 -07:00
Eric Yang
8d150067e2 YARN-9396. Fixed duplicated RM Container created event to ATS. 
Contributed by Prabhu Joseph
 2019-04-04 13:01:56 -04:00
Vrushali C
eb03f7c419 YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph. 2019-04-03 22:53:05 -07:00
Sunil G
002dcc4ebf YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 2019-04-03 18:57:28 +05:30
Yufei Gu
2f752830ba YARN-9214. Add AbstractYarnScheduler#getValidQueues method to remove duplication. Contributed by Wanqiang Ji. 2019-04-01 20:05:15 -07:00
Giovanni Matteo Fumarola
ab2bda57bd YARN-9428. Add metrics for paused containers in NodeManager. Contributed by Abhishek Modi. 2019-04-01 14:21:17 -07:00
Giovanni Matteo Fumarola
da7f8c244d YARN-9431. Fix flaky junit test fair.TestAppRunnability after YARN-8967. Contributed by Wilfred Spiegelenburg. 2019-04-01 11:21:31 -07:00
Giovanni Matteo Fumarola
332cab5518 YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. 2019-04-01 11:06:51 -07:00
Devaraj K
56f1e131ec YARN-9270. Minor cleanup in TestFpgaDiscoverer. Contributed by Peter Bacsko. 2019-03-29 10:58:56 -07:00
Devaraj K
a4cd75e09c YARN-9269. Minor cleanup in FpgaResourceAllocator. Contributed by Peter Bacsko. 2019-03-27 10:08:07 -07:00
yufei
5257f50abb YARN-8967. Change FairScheduler to use PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-25 22:47:24 -07:00
Devaraj K
eeda6891e4 YARN-9268. General improvements in FpgaDevice. Contributed by Peter Bacsko. 2019-03-25 13:22:53 -07:00