big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,225 Commits 2 Branches 2 Tags 264 MiB
37c9b7327d
Commit Graph

3964 Commits

Author SHA1 Message Date
Sunil G
805095496d YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R. 2017-09-12 09:19:12 -07:00
Wangda Tan
53df3eac50 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda) 
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
 2017-09-12 09:19:12 -07:00
Wangda Tan
e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Sunil G
a333ba54e3 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Wangda Tan
39240b61a1 YARN-7043. Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
 2017-09-12 09:19:12 -07:00
Sunil G
758b771943 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 2017-09-12 09:19:12 -07:00
Wangda Tan
df3855541a YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
 2017-09-12 09:19:12 -07:00
Wangda Tan
b1fe3a222e YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda) 
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
 2017-09-12 09:19:12 -07:00
Sunil G
a6989af1d9 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Sunil G
a0030c8c6f YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton. 2017-09-12 09:19:12 -07:00
Daniel Templeton
6746f8cadb YARN-6908. ResourceProfilesManagerImpl is missing @Overrides on methods 
(Contributed by Sunil G. via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Sunil G
2b51b262ab YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 2017-09-12 09:19:12 -07:00
Yufei Gu
d5e9939ebb YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 2017-09-12 09:19:11 -07:00
Daniel Templeton
5c4ab4c291 YARN-6935. [YARN-3926] ResourceProfilesManagerImpl.parseResource() has no need of the key parameter 
(Contributed by Manikandan R via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Daniel Templeton
3aeaafecb8 YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Sunil G
4e5632d28e YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang. 2017-09-12 09:19:11 -07:00
Sunil G
ca2808d8bb YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
58da54640e YARN-6445. [YARN-3926] Performance improvements in resource profile branch with respect to SLS. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
dae65f3bef YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G
7805deed48 YARN-5588. [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Arun Suresh
6708ac3301 YARN-5587. Add support for resource profiles. (vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev
c2032e251e YARN-5708. Implement APIs to get resource profiles from the RM. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev
a9b4426302 YARN-5707. Add manager class for resource profiles. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S
239c1824a0 YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S
9e4ba6aff5 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev
759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Arun Suresh
7ba698997b YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev
ceb12c59a3 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Wangda Tan
32c91223f1 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 2017-09-12 09:19:09 -07:00
Wangda Tan
1bbab7c157 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 2017-09-12 09:19:09 -07:00
Wangda Tan
e74d1be04b YARN-7173. Container update RM-NM communication fix for backward compatibility. (Arun Suresh via wangda) 
Change-Id: I1c39ed5c59dee739ba5044b61b3ef5ed203b79c1
 2017-09-11 20:52:08 -07:00
Haibo Chen
fa531788fd YARN-7132. FairScheduler.initScheduler() contains a surprising unary plus. (Yeliang Cang via Haibo Chen) 2017-09-11 20:42:49 -07:00
Haibo Chen
661f5eb0c6 YARN-7128. The error message in TimelineSchemaCreator is not enough to find out the error. (Jinjiang Ling via Haibo Chen) 2017-09-11 16:20:20 -07:00
Daniel Templeton
6651cbcc72 YARN-6622. Document Docker work as experimental (Contributed by Varun Vasudev) 2017-09-11 16:14:31 -07:00
Daniel Templeton
07951b586e Revert "YARN-6022. Document Docker work as experimental (Contributed by Varun Vasudev)" 
This reverts commit 66be6c848b.
 2017-09-11 16:14:18 -07:00
Daniel Templeton
66be6c848b YARN-6022. Document Docker work as experimental (Contributed by Varun Vasudev) 2017-09-11 15:46:23 -07:00
Yufei Gu
5ab21dfe9c YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu) 2017-09-10 23:17:59 -07:00
Sunil G
722ee84194 YARN-7163. RMContext need not to be injected to webapp and other Always Running services. Contributed by Rohith Sharma K S. 2017-09-11 10:28:10 +05:30
Junping Du
3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
Haibo Chen
8edc60531f HADOOP-14847. Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc. (Bharat Viswanadham via Haibo Chen) 2017-09-08 14:33:11 -07:00
Arun Suresh
1f53ae7972 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 2017-09-08 09:24:05 -07:00
Rohith Sharma K S
4a83170be4 YARN-7140. CollectorInfo should have Public visibility. Contributed by Varun Saxena. 2017-09-08 20:23:16 +05:30
Rohith Sharma K S
0bfb3a2566 YARN-7130. ATSv2 documentation changes post merge. Contributed by Varun Saxena. 2017-09-08 20:21:50 +05:30
Sunil G
ab8368d2e0 YARN-7104. Improve Nodes Heatmap in new YARN UI with better color coding. Contributed by Da Ding. 2017-09-08 16:16:01 +05:30
Subru Krishnan
fa6137501c YARN-5330. SharingPolicy enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-09-07 19:07:17 -07:00
Sunil G
56d93d2e39 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 07:15:17 +05:30
Jason Lowe
b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Wangda Tan
f155ab7cfa YARN-7033. Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan) 
Change-Id: Iffd18bb95debe1c8cc55e30abc1d8f663e9d0e30
 2017-09-07 14:13:37 -07:00
Arun Suresh
c41118a7f8 YARN-6978. Add updateContainer API to NMClient. (Kartheek Muthyala via asuresh) 2017-09-07 10:23:12 -07:00
Sunil G
6f101e7df1 YARN-6992. Kill application button is visible even if the application is FINISHED in RM UI. Contributed by Suma Shivaprasad. 2017-09-07 13:16:20 +05:30
Subru Krishnan
b6e7d13696 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-06 16:46:01 -07:00
Subru Krishnan
e3345e985b Revert "Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem." 
This reverts commit 7996eca7dc.
 2017-09-06 16:39:23 -07:00
Junping Du
22de9449f8 YARN-7144. Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Contributed by Xuan Gong. 2017-09-06 14:53:31 -07:00
Jason Lowe
704267cb49 YARN-7164. TestAMRMClientOnRMRestart fails sporadically with bind address in use. Contributed by Jason Lowe 2017-09-06 15:23:49 -05:00
Junping Du
63231a2a30 YARN-7148. TestLogsCLI fails in trunk and branch-2 and javadoc error. Contributed by Xuan Gong. 2017-09-06 13:19:34 -07:00
Sunil G
ed162b7aa0 YARN-7022. Improve click interaction in queue topology in new YARN UI. Contributed by Abdullah Yousufi. 2017-09-05 08:06:43 +05:30
Varun Saxena
ef87d34a0e YARN-7152. [ATSv2] Registering timeline client before AMRMClient service init throw exception (Rohith Sharma K S via Varun Saxena) 2017-09-04 20:11:02 +05:30
Sunil G
275980bb1e YARN-7075. Better styling for donut charts in new YARN UI. Contributed by Da Ding. 2017-09-02 08:08:59 +05:30
Subru Krishnan
7996eca7dc Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-01 15:16:40 -07:00
Oleg Danilov
c5281a85e1
YARN-7023. Incorrect ReservationId.compareTo() implementation. 
This closes #262

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2017-09-01 21:33:31 +02:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Jason Lowe
99a7f5d451 YARN-7120. CapacitySchedulerPage NPE in "Aggregate scheduler counts" section. Contributed by Eric Payne 2017-09-01 11:09:14 -05:00
Sunil G
a756704f5a YARN-7071. Add vcores and number of containers in new YARN UI node heat map. Contributed by Abdullah Yousufi. 2017-09-01 20:46:39 +05:30
bibinchundatt
dcd0bedcc8 YARN-7141. Move logging APIs to slf4j in timelineservice after ATSv2 merge. Contributed by Varun Saxena 2017-09-01 11:29:16 +05:30
Allen Wittenauer
0adc3a0533
YARN-6721. container-executor should have stack checking 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-31 19:39:31 -07:00
Allen Wittenauer
190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build 
Signed-off-by: Andrew Wang <wang@apache.org>
 2017-08-31 19:06:49 -07:00
Junping Du
91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Subru Krishnan
bac4e8cca8 YARN-7095. Federation: routing getNode/getNodes/getMetrics REST invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 2017-08-31 15:05:41 -07:00
Daniel Templeton
d4417dae4f YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton) 2017-08-31 09:35:01 -07:00
Sunil G
f9e0cc8cdc YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan. 2017-08-31 18:42:01 +05:30
Allen Wittenauer
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Robert Kanter
ce79f7be29 YARN-7094. Document the current known issue with server-side NM graceful decom (rkanter) 2017-08-30 18:04:55 -07:00
Daniel Templeton
cc23514aba YARN-7115. Move BoundedAppender to org.hadoop.yarn.util pacakge 
(Contributed by Jian He via Daniel Templeton)
 2017-08-30 17:26:13 -07:00
Haibo Chen
a20e7105ea YARN-6868. Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml. (Ray Chiang via Haibo Chen) 2017-08-30 09:14:59 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Varun Saxena
512068a01f YARN-6905. Addendum to fix TestTimelineReaderWebServicesHBaseStorage due to missing FastNumberFormat 2017-08-30 11:29:54 +05:30
Varun Saxena
16ba4f544f MAPREDUCE-6838. Addendum to fix code comment 2017-08-30 11:29:54 +05:30
Jian He
08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Vrushali C
bea3e4df76 YARN-6047 Documentation updates for TimelineService v2 (Contributed by Rohith Sharma) 2017-08-30 11:29:54 +05:30
Haibo Chen
9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Varun Saxena
b2efebdd07 YARN-6861. Reader API for sub application entities (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:54 +05:30
Jian He
7fd6ae2479 YARN-6134. [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Rohith Sharma K S
e276c75ec1 YARN-7041. Nodemanager NPE running jobs with security off. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Rohith Sharma K S
32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jian He
b664569586 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jason Lowe
d5ff965fee YARN-6820. Restrict read access to timelineservice v2 data. Contributed by Vrushali C 2017-08-30 11:29:53 +05:30
Varun Saxena
60765aff9b YARN-6874. Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite (Vrushali C via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
354be99dbf YARN-6133. [ATSv2 Security] Renew delegation token for app automatically if an app collector is active. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Varun Saxena
9f6540535d YARN-6734. Ensure sub-application user is extracted & sent to timeline service (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
3fb71b1393 YARN-6888. Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
a990ff70c2 YARN-6733. Add table for storing sub-application entities. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Vrushali C
61136d03f2 YARN-6850 Ensure that supplemented timestamp is stored only for flow run metrics (Contributed by Varun Saxena via Vrushali C) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
70078e91e3 YARN-4455. Support fetching metrics by time range. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S
660413165a YARN-6801. NPE in RM while setting collectors map in NodeHeartbeatResponse. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Jian He
ac7f52df83 YARN-5648. [ATSv2 Security] Client side changes for authentication. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He
d3f11e3f13 YARN-6638. [ATSv2 Security] Timeline reader side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He
879de51206 YARN-5647. [ATSv2 Security] Collector side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
vrushali
24447b3626 YARN-6316 Provide help information and documentation for TimelineSchemaCreator (Contributed by Haibo Chen via Vrushali C) 2017-08-30 11:29:52 +05:30
Varun Saxena
f6a51dce18 YARN-6658. Remove columnFor() methods of Columns in HBaseTimeline backend (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Haibo Chen
a8f082a180 YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 47474fffac)
 2017-08-30 11:29:52 +05:30
Varun Saxena
b87b72b40a YARN-6146. Add Builder methods for TimelineEntityFilters (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena
44999aabc2 YARN-6318. timeline service schema creator fails if executed from a remote machine (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena
18b3a80df7 YARN-6237. Move UID constant to TimelineReaderUtils (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena
c3bd8d6ad3 YARN-6256. Add FROM_ID info key for timeline entities in reader response (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena
8bb2646595 YARN-6027. Support fromid(offset) filter for /flows API (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Sangjin Lee
cf30b3b914 YARN-6253. FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Varun Saxena
ff43b8db7a YARN-6069. CORS support in timeline v2 (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Sangjin Lee
ccb38c1991 YARN-5928. Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Sangjin Lee
580d884913 YARN-6094. Update the coprocessor to be a dynamically loaded one. Contributed by Vrushali C. 2017-08-30 11:29:52 +05:30
Varun Saxena
5e0acee75e Addendum for YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's 2017-08-30 11:29:52 +05:30
Varun Saxena
6f65cf27bb YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena
71847ed44d YARN-5378. Accommodate app-id->cluster mapping (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee
7f54ac48c6 YARN-5980. Update documentation for single node hbase deploy. Contributed by Vrushali C. 2017-08-30 11:29:51 +05:30
Varun Saxena
02a9710a09 YARN-5585. [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee
2556c012e3 YARN-5925. Extract hbase-backend-exclusive utility methods from TimelineStorageUtil. Contributed by Haibo Chen. 2017-08-30 11:29:51 +05:30
Sangjin Lee
ecbc8d4816 YARN-5922. Remove direct references of HBaseTimelineWriter/Reader in core ATS classes. Contributed by Haibo Chen. 
(cherry picked from commit a5a55a54ab)
 2017-08-30 11:29:51 +05:30
Sangjin Lee
4481561e4a YARN-5739. Provide timeline reader API to list available timeline entity types for one application. Contributed by Li Lu. 2017-08-30 11:29:51 +05:30
Sangjin Lee
092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Varun Saxena
c92a7ab31c YARN-3359. Addendum for Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena
91d3443806 YARN-3359. Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee
643a20a358 YARN-5265. Make HBase configuration for the timeline service configurable. Contributed by Joep Rottinghuis. 2017-08-30 11:29:51 +05:30
Sangjin Lee
05ff04439e YARN-5715. Introduce entity prefix for return and sort order. Contributed by Rohith Sharma K S. 2017-08-30 11:29:51 +05:30
Varun Saxena
9bb5cc3d49 YARN-3649. Allow configurable prefix for hbase table names like prod, exp, test etc (Vrushali C via Varun Saxena) 2017-08-30 11:29:50 +05:30
Sangjin Lee
78b7e070d8 YARN-5638. Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery. Contributed by Li Lu. 2017-08-30 11:29:50 +05:30
Li Lu
e72e6260da YARN-5229. Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl. (Vrushali C via gtcarrera9) 2017-08-30 11:29:50 +05:30
Sangjin Lee
278bd6fcff YARN-5156. YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state (Vrushali C via sjlee) 2017-08-30 11:29:50 +05:30
Rohith Sharma K S
4cae120c61 YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null before app is completed. Contributed by Manikandan R. 2017-08-30 10:45:11 +05:30
Jian He
8201ed8009 YARN-6756. ContainerRequest#executionTypeRequest causes NPE. Contributed by Jian He 2017-08-29 20:35:53 -07:00
Wangda Tan
f59332b97b YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda) 
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
 2017-08-29 15:29:22 -07:00
Junping Du
ad45d19998 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang. 2017-08-29 15:16:23 -07:00
Carlo Curino
cc8893edc0 YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-29 14:53:09 -07:00
Jason Lowe
1f6dc4ee9a YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 09:52:24 -05:00
Sunil G
7ddc7aa26e YARN-6386. Show decommissioning nodes in new YARN UI. Contributed by Elek Marton. 2017-08-29 16:51:14 +05:30
Junping Du
312b1fd9da YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He. 2017-08-28 18:26:51 -07:00
Wangda Tan
51881a863c YARN-7112. TestAMRMProxy is failing with invalid request. (Jason Lowe via wangda) 
Change-Id: I432f4523921666ac218fcf9936f826a0d8d635e0
 2017-08-28 11:37:35 -07:00
Yufei Gu
d5bc958279 YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu) 2017-08-28 10:09:46 -07:00
Sunil G
02599bda04 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne. 2017-08-28 12:22:56 +05:30
Li Lu
3bee7046c5 YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9) 2017-08-25 22:43:25 -07:00
Eric Payne
e864f81471 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe. 2017-08-25 15:37:54 -05:00
Jason Lowe
39a9dc8e4a YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne 2017-08-25 15:31:29 -05:00
Arpit Agarwal
8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Jason Lowe
3a4e861169 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2017-08-25 09:16:17 -05:00
Yufei Gu
9e2699ac2c YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu) 2017-08-24 17:52:41 -07:00
Junping Du
c2cb7ea1ef YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 2017-08-24 13:36:49 -07:00
bibinchundatt
de0cba700b YARN-7074. Fix NM state store update comment. Contributed by Botong Huang 2017-08-24 16:34:38 +05:30
Junping Du
652dd434d9 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan. 2017-08-23 18:06:29 -07:00
Wangda Tan
f49843a988 YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda) 
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
 2017-08-23 09:56:20 -07:00
Subru Krishnan
4249172e14 YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 2017-08-22 19:20:57 -07:00
Jason Lowe
657dd59cc8 YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp 2017-08-22 13:16:24 -05:00
Jason Lowe
3efcd51c3b YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles 2017-08-22 12:56:09 -05:00
Akira Ajisaka
d5ff57a08f
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-22 17:14:12 +09:00
Arun Suresh
ae8fb13b31 YARN-6923. Metrics for Federation Router. (Giovanni Matteo Fumarola via asuresh) 2017-08-21 22:50:24 -07:00
Arun Suresh
75abc9a8e2 YARN-5603. Metrics for Federation StateStore. (Ellen Hui via asuresh) 2017-08-21 22:43:08 -07:00
Arun Suresh
7a82d7bcea YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh) 2017-08-20 10:24:05 -07:00
Arun Suresh
8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Wangda Tan
436c2638f9 YARN-6852. Native code changes to support isolate GPU devices by using CGroups. (wangda) 
Change-Id: I4869cc4d8ad539539ccba4bea5a178cacdb741ab
 2017-08-18 18:26:36 -07:00
Yufei Gu
8991f0baec YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu) 2017-08-18 14:38:44 -07:00
bibinchundatt
e05fa3451d YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su. 2017-08-18 20:28:50 +05:30
Jason Lowe
ab1a8ae85f YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-17 15:50:14 -05:00
Sunil G
f9a0e23381 YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad. 2017-08-17 15:07:15 +05:30
Subru Krishnan
de462da04e YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 2017-08-16 11:43:24 -07:00
Jason Lowe
14553061be YARN-7020. TestAMRMProxy#testAMRMProxyTokenRenewal is flakey. Contributed by Robert Kanter 2017-08-16 13:04:36 -05:00
Akira Ajisaka
588c190afd
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka. 2017-08-16 14:06:22 +09:00
Subru Krishnan
75dd866bfb HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 2017-08-15 16:53:59 -07:00
Nathan Roberts
d265459024 YARN-7014. Fix off-by-one error causing heap corruption (Jason Lowe via nroberts) 2017-08-15 15:52:48 -05:00
Sunil G
dadb0c2225 YARN-5146. Support for Fair Scheduler in new YARN UI. Contributed by Abdullah Yousufi. 2017-08-15 21:58:44 +05:30
Arun Suresh
4d7be1d857 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh) 2017-08-14 19:46:17 -07:00
Jason Lowe
5558792894 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne 2017-08-14 15:31:34 -05:00
Jason Lowe
3325ef653d YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles 2017-08-14 14:40:08 -05:00
Daniel Templeton
6b09c32705 YARN-6881. LOG is unused in AllocationConfiguration (Contributed by weiyuan via Daniel Templeton) 2017-08-14 11:55:33 -07:00
Vrushali C
608a06cca5 YARN-6905 Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat (Contributed by Haibo Chen) 2017-08-14 11:41:11 -07:00
Subru Krishnan
18f3603bce YARN-6996. Change javax.cache library implementation from JSR107 to Apache Geronimo. (Ray Chiang via Subru). 2017-08-14 11:10:00 -07:00
Jian He
e2f6299f6f YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:51:30 -07:00
Lei Xu
ce797a1706 HDFS-12221. Replace xcerces in XmlEditsVisitor. (Ajay Kumar via lei) 2017-08-14 10:27:47 -07:00
bibinchundatt
d8f74c3964 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-14 09:39:00 +05:30
Carlo Curino
28d97b79b6 YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino) 2017-08-11 16:58:04 -07:00
Carlo Curino
cc59b5fb26 YARN-6896. Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-11 15:58:01 -07:00
Daniel Templeton
0996acde6c YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)
 2017-08-11 14:50:46 -07:00
Daniel Templeton
65364defb4 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)
 2017-08-11 14:28:55 -07:00
Daniel Templeton
c7680d4cc4 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)
 2017-08-11 14:22:02 -07:00
Daniel Templeton
218588be77 YARN-6952. Enable scheduling monitor in FS (Contributed by Yufei Gu via Daniel Templeton) 2017-08-11 14:04:19 -07:00
Subru Krishnan
bbbf0e2a41 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-11 13:58:45 -07:00
Karthik Kambatla
8c4b6d16a5 YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha) 2017-08-11 12:15:43 -07:00
Sunil G
8d953c2359 YARN-6631. Refactor loader.js in new Yarn UI. Contributed by Akhil P B. 2017-08-10 11:53:26 +05:30
Wangda Tan
ec694145cf YARN-6033. Add support for sections in container-executor configuration file. (Varun Vasudev via wandga) 
Change-Id: Ibc6d2a959debe5d8ff2b51504149742449d1f1da
 2017-08-09 10:51:29 -07:00
Akira Ajisaka
63cfcb90ac
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-09 23:58:22 +09:00
Akira Ajisaka
1a18d5e514
YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 2017-08-09 21:56:43 +09:00
Subru Krishnan
ad2a350662 YARN-6970. Add PoolInitializationException as retriable exception in FederationFacade. (Giovanni Matteo Fumarola via Subru). 2017-08-08 16:48:29 -07:00
Daniel Templeton
e0c24145d2 YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)
 2017-08-08 13:35:22 -07:00
Wangda Tan
1794de3ea4 YARN-6726. Fix issues with docker commands executed by container-executor. (Shane Kumpf via wangda) 
Change-Id: If1b1827345f98f0a49cc7e39d1ba41fbeed5e911
 2017-08-08 12:56:29 -07:00
Jian He
acf9bd8b1d YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:09:38 -07:00
Daniel Templeton
47b145b9b4 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-08-08 10:33:26 -07:00
Akira Ajisaka
98912950b6
YARN-6961. Remove commons-logging dependency from hadoop-yarn-server-applicationhistoryservice module. Contributed by Yeliang Cang. 2017-08-08 19:38:58 +09:00
Arun Suresh
8d3fd81980 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh) 2017-08-07 18:59:25 -07:00
Subru Krishnan
c61f2c4198 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 2017-08-07 16:58:29 -07:00
Wangda Tan
adb84f34db YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
 2017-08-07 11:32:21 -07:00
Akira Ajisaka
b0fbf17965
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 2017-08-07 19:25:40 +09:00
Akira Ajisaka
839e077faf
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-07 18:56:00 +09:00
Sunil G
46b7054fa7 YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 2017-08-07 13:15:46 +05:30
Junping Du
f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh
35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Subru Krishnan
0542e6f86e YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 2017-08-03 14:18:03 -07:00
Yufei Gu
c617fe02b3 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:33:37 -07:00
Haibo Chen
293c74a81b YARN-6673 Add cpu cgroup configurations for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:56:51 -07:00
Haibo Chen
c5d256c760 YARN-6674 Add memory cgroup settings for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:52:35 -07:00
Sunil G
f64cfeaf61 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2017-08-03 19:27:10 +05:30
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Eric Payne
8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu
45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne
48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He
f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino
9625a030de YARN-6853. Add MySql Scripts for FederationStateStore. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit 874ddbf0b5b1d34aca70ee7fc303cbffdde67236)
 2017-08-01 17:28:28 -07:00
Carlo Curino
acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec860819b080ccc82454480a436c0adc)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed26809e12dd62cb66d258ef51c66db3be0a)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5f13c62e8fc45b0af794b5bbeb577da)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
3e1dc7ece0 YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0c1efc52005b1c069d52910d88038d9)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
825108d08a YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd769065eed83193b673ec5d919d69500)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
f427e4201f YARN-6807. Adding required missing configs to Federation configuration guide based on e2e testing. (Tanuj Nayak via Subru). 
(cherry picked from commit b4ac9d1b63dd4031eee8b17d2462087721050b9a)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
4cfec943b1 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de59697cb78686f0e605dc7e93628b5f3297)
 2017-08-01 17:28:26 -07:00
Arun Suresh
433ee44b58 YARN-6691. Update YARN daemon startup/shutdown scripts to include Router service. (Giovanni Matteo Fumarola via asuresh). 
(cherry picked from commit 790cba7ab949c665083c4a9702bb1cd4b7824e8d)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
70b1a757f1 YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235eaf0972783985b1ab24680d029aea79)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
bed1832c93 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200609246cb623c74450ca4a2032063cc)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
d6c29b2f5b YARN-6484. Documenting the YARN Federation feature. (Carlo Curino via Subru). 
(cherry picked from commit 749ca890e6e62671b4df69d020396ec93edadf16)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
42949a6e01 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a7e2fb4b33a7c876eaa4d9122a1deea)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
f317e0cb48 Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec3293fe338553c0d5c3612e6176fb26)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
859aa1f9d6 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
 2017-08-01 17:28:24 -07:00
Carlo Curino
5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
 2017-08-01 17:28:24 -07:00
Carlo Curino
c4bbdfeb66 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29ac44289afad9a15e8737397ede298b)
 2017-08-01 17:28:23 -07:00
Subru Krishnan
e1da8f0667 YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde1cf266ecc7d3b513cdaac6abcebbe4)
 2017-08-01 17:28:23 -07:00
Carlo Curino
1c64e1709b YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c32b178c48c3a01b5a15e4e3a3e750f5)
 2017-08-01 17:28:23 -07:00
Subru Krishnan
86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
 2017-08-01 17:28:23 -07:00
Carlo Curino
8623644f45 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd3a175b122ef86275c99b72964f2c50)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
193ec456d6 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b08de7d16a86f8025221f6f5e0dbcc5)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
98b45b0ed3 YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6155d29435d7c92fd6d68079c4cab86)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
91803305e5 YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade059ff9b38da0aa160f56653769bd0143)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
78032e51d9 YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed51aac4ac9093ac362f6ce64601884d)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
083a05bc14 YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bddb40fbaa77d3f6fec8c99f5589177f)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
93d9fdeca6 YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682eced98dfba55d88edd63296993087c85)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
1dadd0b45a YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522d3504598fb94eee288d54df73418c6)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
0662996b6a YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bdad60a961441dc7e25cdc2553c119f33)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
81472778d7 YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b536f29aa65af71d20c3b3e3765679fd)
 2017-08-01 17:28:20 -07:00
Jian He
2797507d51 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc65090162b3517b194b5e655ee4a157)
 2017-08-01 17:28:20 -07:00
Subru Krishnan
1882bc10fb YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a30427263f4a2eed2018515b2f759fa591)
 2017-08-01 17:28:20 -07:00
Jian He
94a24567d6 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e27623155c715a632d56a68f23523c72)
 2017-08-01 17:28:20 -07:00
Jian He
bdfad4523f YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f8c8abc5fe84efcd05ee0f7dfdd921c)
 2017-08-01 17:28:20 -07:00
Subru Krishnan
cfafd173bd YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70c273ad3371a6f9b458fe0f8a7a6abc)
 2017-08-01 17:28:19 -07:00
Jian He
5c84382397 YARN-3672. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b90ccf1bccb6039edf14140677804c4e)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
52558df620 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9cb2a0d57131e4b7960f47a8dac7940c)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
0733088c29 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41f08dabe4f3a486a2dbd4bed5723867)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
e0c3a44396 YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d0211a757439289bea84d502bed6f6ad52)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
f1a508bdef YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a17ec36547625dff23cc65c9aa89c58e)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
047a16bf15 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042a336a5789b72f0fb6d5f4636d64d49)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
d19b677301 YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a935c2af4ec3f0312ff0c26918c408467)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
dcf8c52d80 YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91bf841e0fee32522ee7b560a6f2598f)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
3e22896f86 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899cd68cb728f60adcefd2a95877cddbca)
 2017-08-01 17:28:17 -07:00
Subru Krishnan
dd75ffb26b YARN-5300. Exclude generated federation protobuf sources from YARN Javadoc/findbugs build 
(cherry picked from commit 9d23570172ae7984f894406dfc6ad066648c61f4)
 2017-08-01 17:28:17 -07:00
Jian He
91f120f743 YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 09:56:33 -07:00
Akira Ajisaka
b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Akira Ajisaka
a4aa1cb405
Revert "YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang." 
This reverts commit 1a78c0ff01.
 2017-08-01 12:15:03 +09:00
Akira Ajisaka
1a78c0ff01
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-01 10:53:32 +09:00
Akira Ajisaka
f8bed5e9a7
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 2017-07-31 11:09:13 +09:00
Arun Suresh
890e14c02a YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 2017-07-28 16:32:43 -07:00
Subru Krishnan
bcde66bed1 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-07-28 15:44:36 -07:00
Yufei Gu
9ea01fd956 YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-07-28 09:52:53 -07:00
Yufei Gu
9902be72cb YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 2017-07-27 23:19:39 -07:00
Akira Ajisaka
38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena
11ece0bda1 Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-28 00:10:45 +05:30
Varun Saxena
f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Daniel Templeton
f81a4efb8c YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 2017-07-25 13:00:31 -07:00
Jian He
ac9489f7fc YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-25 09:57:14 -07:00
Akira Ajisaka
cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Akira Ajisaka
218b1b33ff
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 2017-07-25 15:12:08 +09:00
Daniel Templeton
10583625c1 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-07-24 13:44:00 -07:00
Daniel Templeton
bb30bd3771 YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)
 2017-07-24 12:13:50 -07:00
Daniel Templeton
4c40cd451c YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error message 
(Contributed by Manikandan R via Daniel Templeton)
 2017-07-24 12:03:02 -07:00
Jason Lowe
24853bf32a YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 2017-07-24 10:19:20 -05:00
Sunil G
e315328428 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 11:39:03 +05:30
Brahma Reddy Battula
2054324d47 Revert "YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi" 
This reverts commit 4a771d9010.
 2017-07-23 12:56:18 +08:00
Jian He
4a771d9010 YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-21 11:31:33 -07:00
Jason Lowe
c8df3668ec YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 2017-07-20 11:03:04 -05:00
Arun Suresh
077fcf6a96 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 2017-07-19 12:26:40 -07:00
Rohith Sharma K S
845c4e52bd YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 2017-07-19 11:10:52 +05:30
Yufei Gu
daaf530fce YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 2017-07-18 16:38:07 -07:00
Ray Chiang
f5f14a2ad6 YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 2017-07-18 12:35:08 -07:00
Sunil G
0b7afc060c Addendum patch for YARN-5731 2017-07-18 11:49:09 +05:30
Arun Suresh
5b007921cd YARN-6706. Refactor ContainerScheduler to make oversubscription change easier. (Haibo Chen via asuresh) 2017-07-17 14:11:14 -07:00
Yufei Gu
9e0cde1469 YARN-6625. yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment. (Yufei Gu) 2017-07-14 14:10:45 -07:00
Jason Lowe
a5ae5ac50e YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 2017-07-14 14:56:00 -05:00
bibinchundatt
75c0220b44 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 2017-07-14 13:53:39 +05:30
Yufei Gu
4a574e9a84 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 2017-07-13 23:10:10 -07:00
Sunil G
228ddaa31d YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 2017-07-14 08:07:05 +05:30
Jason Lowe
ebc048cc05 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:44:47 -05:00
Jason Lowe
0ffca5d347 Revert "YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe" 
This reverts commit f76f5c0919.
 2017-07-13 17:42:38 -05:00
Jason Lowe
f76f5c0919 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:38:17 -05:00