big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,477 Commits 2 Branches 2 Tags 264 MiB
8991f0baec
Commit Graph

3770 Commits

Author SHA1 Message Date
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Eric Payne
8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu
45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne
48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He
f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino
9625a030de YARN-6853. Add MySql Scripts for FederationStateStore. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit 874ddbf0b5b1d34aca70ee7fc303cbffdde67236)
 2017-08-01 17:28:28 -07:00
Carlo Curino
acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec860819b080ccc82454480a436c0adc)
 2017-08-01 17:28:28 -07:00
Subru Krishnan
8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed26809e12dd62cb66d258ef51c66db3be0a)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5f13c62e8fc45b0af794b5bbeb577da)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
3e1dc7ece0 YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0c1efc52005b1c069d52910d88038d9)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
825108d08a YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd769065eed83193b673ec5d919d69500)
 2017-08-01 17:28:27 -07:00
Subru Krishnan
f427e4201f YARN-6807. Adding required missing configs to Federation configuration guide based on e2e testing. (Tanuj Nayak via Subru). 
(cherry picked from commit b4ac9d1b63dd4031eee8b17d2462087721050b9a)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
4cfec943b1 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de59697cb78686f0e605dc7e93628b5f3297)
 2017-08-01 17:28:26 -07:00
Arun Suresh
433ee44b58 YARN-6691. Update YARN daemon startup/shutdown scripts to include Router service. (Giovanni Matteo Fumarola via asuresh). 
(cherry picked from commit 790cba7ab949c665083c4a9702bb1cd4b7824e8d)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
70b1a757f1 YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235eaf0972783985b1ab24680d029aea79)
 2017-08-01 17:28:26 -07:00
Subru Krishnan
bed1832c93 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200609246cb623c74450ca4a2032063cc)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
d6c29b2f5b YARN-6484. Documenting the YARN Federation feature. (Carlo Curino via Subru). 
(cherry picked from commit 749ca890e6e62671b4df69d020396ec93edadf16)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
42949a6e01 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a7e2fb4b33a7c876eaa4d9122a1deea)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
f317e0cb48 Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec3293fe338553c0d5c3612e6176fb26)
 2017-08-01 17:28:25 -07:00
Subru Krishnan
859aa1f9d6 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
 2017-08-01 17:28:24 -07:00
Subru Krishnan
80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
 2017-08-01 17:28:24 -07:00
Carlo Curino
5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
 2017-08-01 17:28:24 -07:00
Carlo Curino
c4bbdfeb66 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29ac44289afad9a15e8737397ede298b)
 2017-08-01 17:28:23 -07:00
Subru Krishnan
e1da8f0667 YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde1cf266ecc7d3b513cdaac6abcebbe4)
 2017-08-01 17:28:23 -07:00
Carlo Curino
1c64e1709b YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c32b178c48c3a01b5a15e4e3a3e750f5)
 2017-08-01 17:28:23 -07:00
Subru Krishnan
86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
 2017-08-01 17:28:23 -07:00
Carlo Curino
8623644f45 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd3a175b122ef86275c99b72964f2c50)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
193ec456d6 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b08de7d16a86f8025221f6f5e0dbcc5)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
98b45b0ed3 YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6155d29435d7c92fd6d68079c4cab86)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
91803305e5 YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade059ff9b38da0aa160f56653769bd0143)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
78032e51d9 YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed51aac4ac9093ac362f6ce64601884d)
 2017-08-01 17:28:22 -07:00
Subru Krishnan
4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
083a05bc14 YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bddb40fbaa77d3f6fec8c99f5589177f)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
93d9fdeca6 YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682eced98dfba55d88edd63296993087c85)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
1dadd0b45a YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522d3504598fb94eee288d54df73418c6)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
0662996b6a YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bdad60a961441dc7e25cdc2553c119f33)
 2017-08-01 17:28:21 -07:00
Subru Krishnan
81472778d7 YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b536f29aa65af71d20c3b3e3765679fd)
 2017-08-01 17:28:20 -07:00
Jian He
2797507d51 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc65090162b3517b194b5e655ee4a157)
 2017-08-01 17:28:20 -07:00
Subru Krishnan
1882bc10fb YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a30427263f4a2eed2018515b2f759fa591)
 2017-08-01 17:28:20 -07:00
Jian He
94a24567d6 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e27623155c715a632d56a68f23523c72)
 2017-08-01 17:28:20 -07:00
Jian He
bdfad4523f YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f8c8abc5fe84efcd05ee0f7dfdd921c)
 2017-08-01 17:28:20 -07:00
Subru Krishnan
cfafd173bd YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70c273ad3371a6f9b458fe0f8a7a6abc)
 2017-08-01 17:28:19 -07:00
Jian He
5c84382397 YARN-3672. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b90ccf1bccb6039edf14140677804c4e)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
52558df620 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9cb2a0d57131e4b7960f47a8dac7940c)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
0733088c29 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41f08dabe4f3a486a2dbd4bed5723867)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
e0c3a44396 YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d0211a757439289bea84d502bed6f6ad52)
 2017-08-01 17:28:19 -07:00
Subru Krishnan
f1a508bdef YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a17ec36547625dff23cc65c9aa89c58e)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
047a16bf15 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042a336a5789b72f0fb6d5f4636d64d49)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
d19b677301 YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a935c2af4ec3f0312ff0c26918c408467)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
dcf8c52d80 YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91bf841e0fee32522ee7b560a6f2598f)
 2017-08-01 17:28:18 -07:00
Subru Krishnan
3e22896f86 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899cd68cb728f60adcefd2a95877cddbca)
 2017-08-01 17:28:17 -07:00
Subru Krishnan
dd75ffb26b YARN-5300. Exclude generated federation protobuf sources from YARN Javadoc/findbugs build 
(cherry picked from commit 9d23570172ae7984f894406dfc6ad066648c61f4)
 2017-08-01 17:28:17 -07:00
Jian He
91f120f743 YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 09:56:33 -07:00
Akira Ajisaka
b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Akira Ajisaka
a4aa1cb405
Revert "YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang." 
This reverts commit 1a78c0ff01.
 2017-08-01 12:15:03 +09:00
Akira Ajisaka
1a78c0ff01
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-01 10:53:32 +09:00
Akira Ajisaka
f8bed5e9a7
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 2017-07-31 11:09:13 +09:00
Arun Suresh
890e14c02a YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 2017-07-28 16:32:43 -07:00
Subru Krishnan
bcde66bed1 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-07-28 15:44:36 -07:00
Yufei Gu
9ea01fd956 YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-07-28 09:52:53 -07:00
Yufei Gu
9902be72cb YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 2017-07-27 23:19:39 -07:00
Akira Ajisaka
38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena
11ece0bda1 Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-28 00:10:45 +05:30
Varun Saxena
f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Daniel Templeton
f81a4efb8c YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 2017-07-25 13:00:31 -07:00
Jian He
ac9489f7fc YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-25 09:57:14 -07:00
Akira Ajisaka
cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Akira Ajisaka
218b1b33ff
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 2017-07-25 15:12:08 +09:00
Daniel Templeton
10583625c1 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-07-24 13:44:00 -07:00
Daniel Templeton
bb30bd3771 YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)
 2017-07-24 12:13:50 -07:00
Daniel Templeton
4c40cd451c YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error message 
(Contributed by Manikandan R via Daniel Templeton)
 2017-07-24 12:03:02 -07:00
Jason Lowe
24853bf32a YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 2017-07-24 10:19:20 -05:00
Sunil G
e315328428 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 11:39:03 +05:30
Brahma Reddy Battula
2054324d47 Revert "YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi" 
This reverts commit 4a771d9010.
 2017-07-23 12:56:18 +08:00
Jian He
4a771d9010 YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-21 11:31:33 -07:00
Jason Lowe
c8df3668ec YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 2017-07-20 11:03:04 -05:00
Arun Suresh
077fcf6a96 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 2017-07-19 12:26:40 -07:00
Rohith Sharma K S
845c4e52bd YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 2017-07-19 11:10:52 +05:30
Yufei Gu
daaf530fce YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 2017-07-18 16:38:07 -07:00
Ray Chiang
f5f14a2ad6 YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 2017-07-18 12:35:08 -07:00
Sunil G
0b7afc060c Addendum patch for YARN-5731 2017-07-18 11:49:09 +05:30
Arun Suresh
5b007921cd YARN-6706. Refactor ContainerScheduler to make oversubscription change easier. (Haibo Chen via asuresh) 2017-07-17 14:11:14 -07:00
Yufei Gu
9e0cde1469 YARN-6625. yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment. (Yufei Gu) 2017-07-14 14:10:45 -07:00
Jason Lowe
a5ae5ac50e YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 2017-07-14 14:56:00 -05:00
bibinchundatt
75c0220b44 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 2017-07-14 13:53:39 +05:30
Yufei Gu
4a574e9a84 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 2017-07-13 23:10:10 -07:00
Sunil G
228ddaa31d YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 2017-07-14 08:07:05 +05:30
Jason Lowe
ebc048cc05 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:44:47 -05:00
Jason Lowe
0ffca5d347 Revert "YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe" 
This reverts commit f76f5c0919.
 2017-07-13 17:42:38 -05:00
Jason Lowe
f76f5c0919 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:38:17 -05:00
Jason Lowe
5f1ee72b0e YARN-6654. RollingLevelDBTimelineStore backwards incompatible after fst upgrade. Contributed by Jonathan Eagles 2017-07-13 17:27:40 -05:00
Wangda Tan
945c0958bb YARN-6775. CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls. (Nathan Roberts via wangda) 
Change-Id: I84ccd54200ccbaae23018ef320028e42b4c3509a
 2017-07-13 10:30:15 -07:00
Sunil G
cf0d0844d6 YARN-5731. Preemption calculation is not accurate when reserved containers are present in queue. Contributed by Wangda Tan. 2017-07-13 16:48:29 +05:30
Xuan
9845bea59c YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:40:45 -07:00
Jason Lowe
655110393b YARN-6797. TimelineWriter does not fully consume the POST response. Contributed by Jason Lowe 2017-07-12 15:52:56 -05:00
Akira Ajisaka
b628d0da51
YARN-6809. Fix typo in ResourceManagerHA.md. Contributed by Yeliang Cang. 2017-07-12 19:38:32 +09:00
Sunil G
34f113df5c YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. Contributed by Tao Yang. 2017-07-11 14:52:44 +05:30