big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,178 Commits 2 Branches 2 Tags 264 MiB
b6fff8c81e
Commit Graph

5316 Commits

Author SHA1 Message Date
Giovanni Matteo Fumarola
12c81610e0 YARN-9505. Add container allocation latency for Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-05-17 12:03:21 -07:00
Eric Yang
fab5b80a36 YARN-9554. Fixed TimelineEntity DAO serialization handling. 
Contributed by Prabhu Joseph
 2019-05-16 16:39:50 -04:00
Giovanni Matteo Fumarola
55bd35921c YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-05-15 11:50:46 -07:00
bibinchundatt
570fa2da20 YARN-9508. YarnConfiguration areNodeLabel enabled is costly in allocation flow. Contributed by Bilwa S T. 2019-05-15 13:30:09 +05:30
bibinchundatt
2de1e30658 YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T. 2019-05-15 13:21:39 +05:30
Sunil G
7d831eca64 YARN-9519. TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config. Contributed by Adam Antal. 2019-05-14 10:48:08 -07:00
Giovanni Matteo Fumarola
206e633946 YARN-9453. Clean up code long if-else chain in ApplicationCLI#run. Contributed by Wanqiang Ji. 2019-05-13 11:11:20 -07:00
Giovanni Matteo Fumarola
29ff7fb140 YARN-9493. Scheduler Page does not display the right page by query string. Contributed by Wanqiang Ji. 2019-05-13 10:57:12 -07:00
Weiwei Yang
1a47c2b7ae YARN-9539.Improve cleanup process of app activities and make some conditions configurable. Contributed by Tao Yang. 2019-05-12 22:31:39 -07:00
Sunil G
64c7f36ab1 YARN-9504. [UI2] Fair scheduler queue view page does not show actual capacity. Contributed by Zoltan Siegl. 2019-05-10 14:27:23 +05:30
Giovanni Matteo Fumarola
1b48100a5e YARN-9522. AppBlock ignores full qualified class name of PseudoAuthenticationHandler. Contributed by Prabhu Joseph. 2019-05-09 14:02:58 -07:00
Giovanni Matteo Fumarola
ec36126346 YARN-9483. DistributedShell does not release container when failed to localize at launch. Contributed by Prabhu Joseph. 2019-05-09 10:59:31 -07:00
Weiwei Yang
90add05caa YARN-9489. Support filtering by request-priorities and allocation-request-ids for query results of app activities. Contributed by Tao Yang. 2019-05-09 21:54:09 +08:00
Eric Yang
0c5fa2e7d9 YARN-9535. Fixed spell check for container-executor in Docker documentation. 
Contributed by Charan Hebri
 2019-05-08 16:56:26 -04:00
Akira Ajisaka
3172f6cbf9
YARN-9513. Addendum patch: Fix ASF License warnings. Contributed by Giovanni Matteo Fumarola. 2019-05-08 14:56:23 +09:00
Weiwei Yang
c336af3847 YARN-9432. Reserved containers leak after its request has been cancelled or satisfied when multi-nodes enabled. Contributed by Tao Yang. 2019-05-08 09:54:16 +08:00
Akira Ajisaka
66c2a4ef89
HADOOP-16270. [JDK 11] Remove unintentional override of the version of Maven Dependency Plugin. Contributed by Xieming Li. 2019-05-08 10:50:07 +09:00
Giovanni Matteo Fumarola
8ecbf61cca YARN-9513. [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8. Contributed by Adam Antal. 2019-05-07 10:59:02 -07:00
Eric Yang
49e1292ea3 YARN-9524. Fixed TestAHSWebService and TestLogsCLI unit tests. 
Contributed by Prabhu Joseph
 2019-05-06 19:48:45 -04:00
Haibo Chen
597fa47ad1 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 2019-05-06 11:56:22 -07:00
Weiwei Yang
12b7059ddc YARN-9440. Improve diagnostics for scheduler and app activities. Contributed by Tao Yang. 2019-05-06 20:00:15 +08:00
Giovanni Matteo Fumarola
d331a2a0c2 YARN-9528. Federation RMs starting up at the same time can give duplicate application IDs. Contributed by Young Chen. 2019-05-03 15:05:17 -07:00
Eric E Payne
b094b94d43 YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 2019-05-02 19:39:26 +00:00
Eric Yang
accb811e57 YARN-6929. Improved partition algorithm for yarn remote-app-log-dir. 
Contributed by Prabhu Joseph
 2019-04-30 17:04:59 -04:00
Zhankun Tang
7fbaa7d66f YARN-9476. [YARN-9473] Create unit tests for VE plugin. Contributed by Peter Bacsko. 2019-04-30 11:06:44 +08:00
Eric Badger
79d3d35398 YARN-9486. Docker container exited with failure does not get clean up correctly. Contributed by Eric Yang 2019-04-26 01:21:28 +00:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Giovanni Matteo Fumarola
3f2f4186f6 YARN-9424. Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent. Contributed by Shen Yinjie. 2019-04-23 19:58:41 -07:00
Akira Ajisaka
9d400627c2 YARN-9081. Update jackson from 1.9.13 to 2.x in hadoop-yarn-services-core. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-04-24 09:50:25 +09:00
Giovanni Matteo Fumarola
fec9bf4b0b YARN-9501. TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 15:42:56 -07:00
Giovanni Matteo Fumarola
4a0ba24959 YARN-9491. TestApplicationMasterServiceFair#ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 15:27:04 -07:00
Inigo Goiri
c504eee0c2 YARN-9339. Apps pending metric incorrect after moving app to a new queue. Contributed by Abhishek Modi. 2019-04-23 12:40:44 -07:00
Zhankun Tang
8a95ea61e1 YARN-9475. [YARN-9473] Create basic VE plugin. Contributed by Peter Bacsko. 2019-04-23 17:33:58 +08:00
Weiwei Yang
1c8046d67e YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph. 2019-04-23 14:21:13 +08:00
Eric Yang
a54c1e3ace YARN-8551. Project setup for MaWo application. 
Contributed by Yesha Vora
 2019-04-22 16:56:26 -04:00
Inigo Goiri
96e3027e46 YARN-2889. Limit the number of opportunistic container allocated per AM heartbeat. Contributed by Abhishek Modi. 2019-04-22 09:49:03 -07:00
Weiwei Yang
adefd37105 YARN-9495. Fix findbugs warnings in hadoop-yarn-server-resourcemanager module. Contributed by Tao Yang. 2019-04-20 21:51:00 +08:00
Billie Rinaldi
317fcbabe3 YARN-9254. Add support for storing application catalog data to HDFS. Contributed by Eric Yang 2019-04-19 10:29:57 -07:00
Inigo Goiri
aeadb9432f YARN-9448. Fix Opportunistic Scheduling for node local allocations. Contributed by Abhishek Modi. 2019-04-19 09:41:06 -07:00
Eric Yang
ef97a20831 YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng
 2019-04-18 18:59:21 -04:00
Eric Yang
df76cdc895 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph
 2019-04-18 12:29:37 -04:00
Prabhu Joseph
aa4c744aef
YARN-9470. Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-18 15:40:37 +09:00
Siyao Meng
6e4399ea61 YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-17 22:56:57 -07:00
Eric Yang
9cf7401794 YARN-9349. Improved log level practices for InvalidStateTransitionException. 
Contributed by Anuhan Torgonshar

(cherry picked from commit fe2370e039e1ee980d74769ae85d67434e0993cf)
 2019-04-16 19:53:45 -04:00
Szilard Nemeth
b8086aed86 YARN-9123. Clean up and split testcases in TestNMWebServices for GPU support. Contributed by Szilard Nemeth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-16 11:06:25 -07:00
Billie Rinaldi
2364c7d0bf YARN-9466. Fixed application catalog navigation bar height in Safari. Contributed by Eric Yang 2019-04-16 10:04:27 -07:00
Billie Rinaldi
ad865888a6 YARN-8530. Add SPNEGO filter to application catalog. Contributed by Eric Yang 2019-04-16 09:52:14 -07:00
Akira Ajisaka
c65218ee8e YARN-8943. Upgrade JUnit from 4 to 5 in hadoop-yarn-api. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-04-16 12:13:49 +09:00
Eric Badger
5583e1b6fc YARN-7848 Force removal of docker containers that do not get removed on first try. Contributed by Eric Yang 2019-04-15 20:47:09 +00:00
Eric Badger
254efc9358 YARN-9379. Can't specify docker runtime through environment. Contributed by caozhiqiang 2019-04-15 18:24:37 +00:00
Weiwei Yang
7fa73fac26 YARN-9439. Support asynchronized scheduling mode and multi-node lookup mechanism for app activities. Contributed by Tao Yang. 2019-04-16 00:12:43 +08:00
Akira Ajisaka
c4c16cae7d
YARN-9481. [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp. Contributed by Kei Kori. 2019-04-15 13:08:28 +09:00
Inigo Goiri
7a68e7abd5 YARN-9474. Remove hard coded sleep from Opportunistic Scheduler tests. Contributed by Abhishek Modi. 2019-04-14 20:11:20 -07:00
Billie Rinaldi
b2cdf809bc YARN-9281. Add express upgrade button to Appcatalog UI. Contributed by Eric Yang 2019-04-13 18:55:11 +03:00
Gabor Bota
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre. 
Author:    Gabor Bota <gabor.bota@cloudera.com>
 2019-04-12 18:28:38 -07:00
Giovanni Matteo Fumarola
ed3747c1cc YARN-9435. Add Opportunistic Scheduler metrics in ResourceManager. Contributed by Abhishek Modi. 2019-04-11 11:49:19 -07:00
Weiwei Yang
8c1bba375b YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 2019-04-10 22:51:28 +08:00
Igor Rudenko
32722d2661
YARN-9433. Remove unused constants in YARN resource manager 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-10 18:37:27 +09:00
Giovanni Matteo Fumarola
cfec455c45 YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Contributed by Inigo Goiri. 2019-04-09 10:59:43 -07:00
Weiwei Yang
fc05b0e70e YARN-9313. Support asynchronized scheduling mode and multi-node lookup mechanism for scheduler activities. Contributed by Tao Yang. 2019-04-08 13:40:53 +08:00
Weiwei Yang
ec143cbf67 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 2019-04-06 19:59:36 +08:00
Vrushali C
22362c876d YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi. 2019-04-05 12:06:51 -07:00
Vrushali C
27039a29ae YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. 2019-04-05 12:02:43 -07:00
Eric Yang
2bf82aee82 YARN-9441. Updated YARN app catalog name for consistency. 
Contributed by Weiwei Yang
 2019-04-04 13:21:30 -04:00
Eric Yang
8d150067e2 YARN-9396. Fixed duplicated RM Container created event to ATS. 
Contributed by Prabhu Joseph
 2019-04-04 13:01:56 -04:00
Weiwei Yang
945e8c6064 YARN-9394. Use new API of RackResolver to get better performance. Contributed by Lantao Jin. 2019-04-04 18:05:01 +08:00
Vrushali C
eb03f7c419 YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph. 2019-04-03 22:53:05 -07:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Sunil G
002dcc4ebf YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 2019-04-03 18:57:28 +05:30
Yufei Gu
2f752830ba YARN-9214. Add AbstractYarnScheduler#getValidQueues method to remove duplication. Contributed by Wanqiang Ji. 2019-04-01 20:05:15 -07:00
Giovanni Matteo Fumarola
ab2bda57bd YARN-9428. Add metrics for paused containers in NodeManager. Contributed by Abhishek Modi. 2019-04-01 14:21:17 -07:00
Giovanni Matteo Fumarola
da7f8c244d YARN-9431. Fix flaky junit test fair.TestAppRunnability after YARN-8967. Contributed by Wilfred Spiegelenburg. 2019-04-01 11:21:31 -07:00
Giovanni Matteo Fumarola
332cab5518 YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. 2019-04-01 11:06:51 -07:00
Giovanni Matteo Fumarola
b0d24ef39c YARN-9227. DistributedShell RelativePath is not removed at end. Contributed by Prabhu Joseph. 2019-04-01 10:59:14 -07:00
Billie Rinaldi
35b0a381e7 YARN-9255. Improve recommend applications order and fix findbugs warnings. Contributed by Eric Yang 2019-04-01 09:21:36 -07:00
Billie Rinaldi
d9e9e5657d YARN-9348. Application catalog build system bug fixes. Contributed by Eric Yang 2019-03-29 13:51:21 -07:00
Billie Rinaldi
7ee32a2eb1 YARN-7129. Application Catalog for YARN applications. Contributed by Eric Yang 2019-03-29 13:51:21 -07:00
Devaraj K
56f1e131ec YARN-9270. Minor cleanup in TestFpgaDiscoverer. Contributed by Peter Bacsko. 2019-03-29 10:58:56 -07:00
Giovanni Matteo Fumarola
9cd66198ee YARN-9411. TestYarnNativeServices fails sporadically with bind address in use. Contributed by Prabhu Joseph. 2019-03-27 10:44:54 -07:00
Devaraj K
a4cd75e09c YARN-9269. Minor cleanup in FpgaResourceAllocator. Contributed by Peter Bacsko. 2019-03-27 10:08:07 -07:00
yufei
5257f50abb YARN-8967. Change FairScheduler to use PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-25 22:47:24 -07:00
Eric Yang
710cbc9bd6 YARN-9405. Fixed flaky tests in TestYarnNativeServices. 
Contributed by Prabhu Joseph
 2019-03-25 16:34:04 -04:00
Devaraj K
eeda6891e4 YARN-9268. General improvements in FpgaDevice. Contributed by Peter Bacsko. 2019-03-25 13:22:53 -07:00
Eric Yang
3c45762a0b YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan
 2019-03-25 15:53:24 -04:00
Giovanni Matteo Fumarola
509b20b292 YARN-9404. TestApplicationLifetimeMonitor#testApplicationLifetimeMonitor fails intermittent. Contributed by Prabhu Joseph. 2019-03-22 11:45:39 -07:00
Zoltan Siegl
ce5eb9cb2e YARN-9358. Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 
(Contributed by Zoltan Siegl via Daniel Templeton)

Change-Id: I92d52c0ca630e71afb26b2b7587cbdbe79254a05
 2019-03-22 12:28:34 +01:00
Giovanni Matteo Fumarola
548997d6c9 YARN-9402. Opportunistic containers should not be scheduled on Decommissioning nodes. Contributed by Abhishek Modi. 2019-03-21 12:04:05 -07:00
Devaraj K
a99eb80659 YARN-9267. General improvements in FpgaResourceHandlerImpl. Contributed by Peter Bacsko. 2019-03-21 11:15:56 -07:00
Eric Yang
506502bb83 YARN-9370. Added logging for recovering assigned GPU devices. 
Contributed by Yesha Vora
 2019-03-20 19:12:19 -04:00
Eric Yang
f2b862cac6 YARN-9398. Fixed javadoc errors for FPGA related java files. 
Contributed by Peter Bacsko
 2019-03-20 15:45:37 -04:00
Rohith Sharma K S
b3b0e332e6 YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph. 2019-03-20 21:24:31 +05:30
Rohith Sharma K S
0d24684eee YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph. 2019-03-20 21:22:54 +05:30
Rohith Sharma K S
f6e60e966e YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph. 2019-03-20 21:20:50 +05:30
Rohith Sharma K S
c1a4eeb7c8 YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph. 2019-03-20 21:18:19 +05:30
Giovanni Matteo Fumarola
5d8bd0e5cb YARN-9392. Handle missing scheduler events in Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-03-19 11:00:21 -07:00
Rohith Sharma K S
c9e50c48de YARN-9365. Fix table creation command in TimelineServiceV2.md 2019-03-19 12:05:37 +05:30
Eric Yang
09eabda314 YARN-9364. Remove commons-logging dependency from YARN. 
Contributed by Prabhu Joseph
 2019-03-18 19:58:42 -04:00
Eric Yang
5f6e225166 YARN-9363. Replaced debug logging with SLF4J parameterized log message. 
Contributed by Prabhu Joseph
 2019-03-18 13:57:18 -04:00
Eric Yang
19b22c4385 YARN-9385. Fixed ApiServiceClient to use current UGI. 
Contributed by Eric Yang
 2019-03-18 13:16:34 -04:00
Shweta Yakkali
0e7e9013d4 YARN-9340. [Clean-up] Remove NULL check before instanceof in ResourceRequestSetKey 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I932e29b36f086f7b7c76a250e33b473617ddbda1
 2019-03-18 15:08:37 +01:00