big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20,040 Commits 2 Branches 2 Tags 264 MiB
2df0a8dcb3
Commit Graph

4811 Commits

Author SHA1 Message Date
bibinchundatt
eed8415dc1 YARN-8535. Fix DistributedShell unit tests. Contributed by Abhishek Modi. 2018-09-02 13:35:52 +05:30
Shane Kumpf
73625168c0 YARN-8642. Add support for tmpfs mounts with the Docker runtime. Contributed by Craig Condit 2018-08-29 07:08:37 -06:00
Weiwei Yang
3fa4639421 YARN-8723. Fix a typo in CS init error message when resource calculator is not correctly set. Contributed by Abhishek Modi. 2018-08-29 11:13:44 +08:00
Giovanni Matteo Fumarola
7ed458b255 YARN-8697. LocalityMulticastAMRMProxyPolicy should fallback to random sub-cluster when cannot resolve resource. Contributed by Botong Huang. 2018-08-28 16:01:35 -07:00
Eric Yang
fd089caf69 YARN-8488. Added SUCCEEDED/FAILED states to YARN service. 
Contributed by Suma Shivaprasad
 2018-08-28 13:55:28 -04:00
Giovanni Matteo Fumarola
602d13844a HADOOP-15699. Fix some of testContainerManager failures in Windows. Contributed by Botong Huang. 2018-08-27 12:25:46 -07:00
Billie Rinaldi
05b2bbeb35 YARN-8675. Remove default hostname for docker containers when net=host. Contributed by Suma Shivaprasad 2018-08-27 11:34:33 -07:00
Giovanni Matteo Fumarola
f152582562 YARN-8705. Refactor the UAM heartbeat thread in preparation for YARN-8696. Contributed by Botong Huang. 2018-08-27 10:32:22 -07:00
Weiwei Yang
e8b063f630 YARN-8719. Typo correction for yarn configuration in OpportunisticContainers(federation) docs. Contributed by Y. SREENIVASULU REDDY. 2018-08-28 01:03:03 +08:00
Jason Lowe
585ebd873a YARN-8649. NPE in localizer hearbeat processing if a container is killed while localizing. Contributed by lujie 2018-08-23 09:29:46 -05:00
Sunil G
1ac01444a2 YARN-8015. Support all types of placement constraint support for Capacity Scheduler. Contributed by Weiwei Yang. 2018-08-23 10:05:43 +05:30
Eric Yang
e557c6bd8d YARN-8298. Added express upgrade for YARN service. 
Contributed by Chandni Singh
 2018-08-21 19:49:26 -04:00
Weiwei Yang
9c3fc3ef28 YARN-7494. Add muti-node lookup mechanism and pluggable nodes sorting policies to optimize placement decision. Contributed by Sunil Govindan. 2018-08-21 22:42:28 +08:00
Weiwei Yang
54d0bf8935 YARN-8683. Support to display pending scheduling requests in RM app attempt page. Contributed by Tao Yang. 2018-08-21 19:00:31 +08:00
Rohith Sharma K S
d3fef7a5c5 YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi. 2018-08-21 11:58:07 +05:30
Giovanni Matteo Fumarola
e0f6ffdbad YARN-8581. [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy. Contributed by Botong Huang. 2018-08-20 14:33:16 -07:00
Giovanni Matteo Fumarola
8736fc39ac YARN-8673. [AMRMProxy] More robust responseId resync after an YarnRM master slave switch. Contributed by Botong Huang. 2018-08-20 12:22:36 -07:00
Jason Lowe
65e7469712 YARN-8242. YARN NM: OOM error while reading back the state store on recovery. Contributed by Pradeep Ambati and Kanwaljeet Sachdev 2018-08-20 10:14:40 -05:00
Rohith Sharma K S
4aacbfff60 YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan. 2018-08-18 10:26:55 +05:30
Rohith Sharma K S
1697a02306 YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli. 2018-08-17 11:12:10 +05:30
Eric Yang
d42806160e YARN-8667. Cleanup symlinks when container restarted by NM. 
Contributed by Chandni Singh
 2018-08-16 18:41:58 -04:00
Eric Yang
8990eaf592 YARN-8474. Fixed ApiServiceClient kerberos negotiation. 
Contributed by Billie Rinaldi
 2018-08-16 12:46:37 -04:00
Jason Lowe
cb21eaa026 YARN-8656. container-executor should not write cgroup tasks files for docker containers. Contributed by Jim Brennan 2018-08-16 10:06:17 -05:00
Weiwei Yang
8dc07b4723 YARN-8614. Fix few annotation typos in YarnConfiguration. Contributed by Sen Zhao. 2018-08-15 12:15:54 +08:00
Wei-Chiu Chuang
bdd0e01ed9 HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 18:25:38 -07:00
Wei-Chiu Chuang
363bd16e31 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 17:19:00 -07:00
Eric Yang
4cba0741d5 YARN-8160. Support upgrade of service that use docker containers. 
Contributed by Chandni Singh
 2018-08-14 11:36:26 -04:00
Jason Lowe
d1d129aa9d YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan 2018-08-14 10:21:03 -05:00
Eric Yang
74411ce0ce YARN-7417. Remove duplicated code from IndexedFileAggregatedLogsBlock 
and TFileAggregatedLogsBlock.
           Contributed by Zian Chen
 2018-08-13 16:50:00 -04:00
Sunil G
cadbc8b57f YARN-8561. [Submarine] Initial implementation: Training job submission and job history retrieval. Contributed by Wangda Tan. 2018-08-13 14:22:55 +05:30
Shane Kumpf
e7951c69cb YARN-8520. Document best practice for user management. Contributed by Eric Yang 2018-08-10 14:32:03 -06:00
Weiwei Yang
0a71bf1452 YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang. 2018-08-10 14:37:45 +08:00
Weiwei Yang
08d5060605 YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang. 2018-08-10 08:32:02 +08:00
Robert Kanter
b2517dd66b YARN-4946. RM should not consider an application as COMPLETED when log aggregation is not in a terminal state (snemeth via rkanter) 2018-08-09 14:58:04 -07:00
Wangda Tan
8244abb7ae YARN-8136. Add version attribute to site doc examples and quickstart. (Eric Yang via wangda) 
Change-Id: I4541b239f490ca0a6edf9698e0d3deaf83669151
 2018-08-09 11:04:02 -07:00
Wangda Tan
344c335a92 YARN-8588. Logging improvements for better debuggability. (Suma Shivaprasad via wangda) 
Change-Id: I66aa4b0ec031ae5ce0fae558e2f8cbcbbfebc442
 2018-08-09 11:04:02 -07:00
Weiwei Yang
d352f167eb YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang. 2018-08-09 23:46:53 +08:00
Jason Lowe
cd04e954d2 YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati 2018-08-09 10:17:34 -05:00
Sunil G
00013d6ef7 YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB. 2018-08-09 12:18:32 +05:30
Robert Kanter
8478732bb2 YARN-8568. Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md (bsteinbach via rkanter) 2018-08-08 15:08:55 -07:00
Wangda Tan
3214cd75ac Make 3.1.1 awared by other branches - adding missing files 2018-08-08 13:05:24 -07:00
Wangda Tan
49c687608b Make 3.1.1 awared by other branches 2018-08-08 13:02:12 -07:00
bibinchundatt
ff06bd1be8 YARN-8601. Print ExecutionType in Container report CLI. Contributed by Bilwa S T. 2018-08-08 22:42:52 +05:30
Sunil G
5b898c176f Revert "YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB." 
This reverts commit 64901abdfa.
 2018-08-08 19:48:49 +05:30
Sunil G
64901abdfa YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB. 2018-08-08 19:43:29 +05:30
Giovanni Matteo Fumarola
d838179d8d YARN-8626. Create HomePolicyManager that sends all the requests to the home subcluster. Contributed by Inigo Goiri. 2018-08-07 15:33:16 -07:00
Wangda Tan
861095f761 YARN-8407. Container launch exception in AM log should be printed in ERROR level. (Yesha Vora via wangda) 
Change-Id: I154e873df1df3503a09c41d6b3874ca195af91d9
 2018-08-07 13:01:13 -07:00
Wangda Tan
c0599151bb YARN-7089. Mark the log-aggregation-controller APIs as public. (Zian Chen via wangda) 
Change-Id: I37851bdc5935d623a27d0973a206c997258716eb
 2018-08-07 12:37:32 -07:00
Wangda Tan
d4258fcad7 YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma Shivaprasad via wangda) 
Change-Id: I392ef4f8baa84d5d7b1f2e438c560b5426b6d4f2
 2018-08-07 12:36:55 -07:00
Eric Yang
ca20e0d7e9 YARN-8624. Updated verbiage around entry point support. 
Contributed by Craig Condit
 2018-08-06 12:37:59 -04:00
Shane Kumpf
bcfc9851f4 YARN-8287. Update documentation and yarn-default related to the Docker runtime. Contributed by Craig Condit 2018-08-03 18:20:49 -06:00
Sunil G
3426f4062a YARN-8615. [UI2] Resource Usage tab shows only memory related info. No info available for vcores/gpu. Contributed by Akhil PB. 2018-08-03 19:28:16 +05:30
Sunil G
29417dbbf4 YARN-8620. [UI2] YARN Services UI new submission failures are not debuggable. Contributed by Akhil PB. 2018-08-03 19:12:34 +05:30
Rohith Sharma K S
022592ae79 YARN-8608. [UI2] No information available per application appAttempt about 'Total Outstanding Resource Requests'. Contributed by Akhil PB. 2018-08-03 15:30:39 +05:30
Rohith Sharma K S
33482d35e8 YARN-8603. [UI2] Latest run application should be listed first in the RM UI. Contributed by Akhil PB. 2018-08-03 15:30:39 +05:30
Haibo Chen
889df6f194 YARN-7948. Enable fair scheduler to refresh maximum allocation for multiple resource types. (Szilard Nemeth via Haibo Chen) 2018-08-02 10:10:12 -07:00
Sunil G
12a095a496 YARN-7159. Normalize unit of resource objects in RM to avoid unit conversion in critical path. Contributed by Manikandan R. 2018-08-02 22:29:21 +05:30
Jason Lowe
7526815e32 YARN-8263. DockerClient still touches hadoop.tmp.dir. Contributed by Craig Condit 2018-08-02 10:43:48 -05:00
Sunil G
5033d7da8f YARN-8318. [UI2] IP address in component page shows N/A. Contributed by Yesha Vora. 2018-08-02 20:09:24 +05:30
Sunil G
97870ec1f6 YARN-8592. [UI2] rmip:port/ui2 endpoint shows a blank page in windows OS and Chrome browser. Contributed by Akhil PB. 2018-08-02 16:10:54 +05:30
Sunil G
1ea81169ba YARN-8594. [UI2] Display current logged in user. Contributed by Akhil PB. 2018-08-02 12:41:06 +05:30
Sunil G
735b492556 YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB. 2018-08-02 08:34:09 +05:30
Eric Yang
23f394240e YARN-8610. Fixed initiate upgrade error message. 
Contributed by Chandni Singh
 2018-08-01 20:41:43 -04:00
Shane Kumpf
603a57476c YARN-8600. RegistryDNS hang when remote lookup does not reply. Contributed by Eric Yang 2018-08-01 12:22:01 -06:00
Billie Rinaldi
67c65da261 YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang 2018-08-01 08:51:40 -07:00
Sunil G
d920b9db77 YARN-8595. [UI2] Container diagnostic information is missing from container page. Contributed by Akhil PB. 2018-08-01 14:27:54 +05:30
Sunil G
a48a0cc7fd YARN-8606. Opportunistic scheduling does not work post RM failover. Contributed by Bibin A Chundatt. 2018-08-01 12:17:18 +05:30
Wangda Tan
5cc8e99147 YARN-8522. Application fails with InvalidResourceRequestException. (Zian Chen via wangda) 
Change-Id: I34dd7fa49bd4d10580c4a78051033b1068d28f1e
 2018-07-31 22:34:53 -07:00
Sunil G
6310c0d17d YARN-8397. Potential thread leak in ActivitiesManager. Contributed by Rohith Sharma K S. 2018-08-01 08:33:30 +05:30
Eric Yang
c7ebcd76bf YARN-8579. Recover NMToken of previous attempted component data. 
Contributed by Gour Saha
 2018-07-31 18:01:02 -04:00
Wangda Tan
4b540bbfcf YARN-8418. App local logs could leaked if log aggregation fails to initialize for the app. (Bibin A Chundatt via wangda) 
Change-Id: I29a23ca4b219b48c92e7975cd44cddb8b0e04104
 2018-07-31 12:08:00 -07:00
Haibo Chen
8aa93a575e YARN-8605. TestDominantResourceFairnessPolicy.testModWhileSorting is flaky. (Wilfred Spiegelenburg via Haibo Chen) 2018-07-31 11:32:40 -07:00
Inigo Goiri
9fea5c9ee7 YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi. 2018-07-31 09:36:34 -07:00
Jonathan Hung
3e06a5dcea YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:44:18 -07:00
bibinchundatt
2b39ad2698 YARN-8584. Several typos in Log Aggregation related classes. Contributed by Szilard Nemeth. 2018-07-30 23:25:19 +05:30
Sunil G
63e08ec071 YARN-8591. [ATSv2] NPE while checking for entity acl in non-secure cluster. Contributed by Rohith Sharma K S. 2018-07-30 14:48:04 +05:30
bibinchundatt
3d586841ab YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt. 2018-07-28 20:52:39 +05:30
Eric Yang
ed9d60e888 YARN-8508. Release GPU resource for killed container. 
Contributed by Chandni Singh
 2018-07-27 19:33:58 -04:00
Inigo Goiri
79091cf76f YARN-8596. Allow SQLFederationStateStore to submit the same app in the same subcluster. Contributed by Giovanni Matteo Fumarola. 2018-07-27 15:23:57 -07:00
Robert Kanter
2cccf4061c YARN-8517. getContainer and getContainers ResourceManager REST API methods are not documented (snemeth via rkanter) 2018-07-27 14:35:03 -07:00
Robert Kanter
fecbac499e YARN-8566. Add diagnostic message for unschedulable containers (snemeth via rkanter) 2018-07-27 14:32:34 -07:00
Billie Rinaldi
b429f19d32 YARN-8571. Validate service principal format prior to launching yarn service. Contributed by Eric Yang 2018-07-27 11:30:19 -07:00
Eric Yang
8d3c068e59 YARN-8429. Improve diagnostic message when artifact is not set properly. 
Contributed by Gour Saha
 2018-07-26 20:02:13 -04:00
Eric Yang
40fad32824 YARN-8545. Return allocated resource to RM for failed container. 
Contributed by Chandni Singh
 2018-07-26 18:22:57 -04:00
Eric Yang
f93ecf5c1e YARN-8330. Improved publishing ALLOCATED events to ATS. 
Contributed by Suma Shivaprasad
 2018-07-25 18:49:30 -04:00
Eric E Payne
9485c9aee6 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R 2018-07-25 16:22:04 +00:00
Weiwei Yang
5be9f4a5d0 YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 2018-07-25 17:35:27 +08:00
Haibo Chen
ea2c6c8c9a YARN-7133. Clean up lock-try order in fair scheduler. (Szilard Nemeth via Haibo Chen) 2018-07-24 12:46:59 -07:00
bibinchundatt
e673dd1d4d YARN-8541. RM startup failure on recovery after user deletion. Contributed by Bibin A Chundatt. 2018-07-24 23:26:59 +05:30
Sunil G
35ce6eb1f5 YARN-7748. TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events. Contributed by Weiwei Yang. 2018-07-24 22:20:17 +05:30
bibinchundatt
ff7c2eda34 YARN-8548. AllocationRespose proto setNMToken initBuilder not done. Contributed by Bilwa S T. 2018-07-24 16:17:20 +05:30
bibinchundatt
8461278833 YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt. 2018-07-24 13:09:17 +05:30
Eric Yang
8688a0c7f8 YARN-8380. Support bind propagation options for mounts in docker runtime. 
Contributed by Billie Rinaldi
 2018-07-23 20:12:04 -04:00
Haibo Chen
9d3c39e9dd YARN-6966. NodeManager metrics may return wrong negative values when NM restart. (Szilard Nemeth via Haibo Chen) 2018-07-23 11:07:24 -07:00
Eric Yang
84d7bf1eef YARN-8360. Improve YARN service restart policy and node manager auto restart policy. 
Contributed by Suma Shivaprasad
 2018-07-23 12:57:01 -04:00
Eric Yang
1622a4b810 YARN-8301. Added YARN service upgrade instructions. 
Contributed by Chandni Singh
 2018-07-20 19:46:35 -04:00
Haibo Chen
8a6bb8409c YARN-6964. Fair scheduler misuses Resources operations. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-20 10:46:28 -07:00
Weiwei Yang
cbf2026483 YARN-8528. Final states in ContainerAllocation might be modified externally causing unexpected allocation results. Contributed by Xintong Song. 2018-07-20 22:34:06 +08:00
Haibo Chen
e6873dfde0 YARN-7300. DiskValidator is not used in LocalDirAllocator. (Szilard Nemeth via Haibo Chen) 2018-07-19 16:27:11 -07:00
Haibo Chen
f354f47f99 YARN-6995. Improve use of ResourceNotFoundException in resource types code. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-19 15:35:05 -07:00