big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13,843 Commits 2 Branches 2 Tags 264 MiB
9560f252cf
Commit Graph

2786 Commits

Author SHA1 Message Date
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Karthik Kambatla
f5ff05cc8a YARN-5132. Exclude generated protobuf sources from YARN Javadoc build. (Subru Krishnan via kasha) 2016-05-28 08:09:46 -07:00
Arun Suresh
4fc09a897b YARN-5117. QueuingContainerManager does not start GUARANTEED Container even if Resources are available. (Konstantinos Karanasos via asuresh) 2016-05-27 17:49:15 -07:00
Arun Suresh
aa975bc781 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh) 2016-05-27 14:06:32 -07:00
Akira Ajisaka
e4022debf7 YARN-5166. javadoc:javadoc goal fails on hadoop-yarn-client. Contributed by Andras Bokor. 2016-05-27 23:15:32 +09:00
Junping Du
705286ccae YARN-4987. Read cache concurrency issue between read and evict in EntityGroupFS timeline store. Contributed by Li Lu. 2016-05-27 06:58:32 -07:00
Xuan
bde819abbb YARN-5089. Improve "yarn log" command-line "logFiles" option to support 
regex. Contributed by Xuan Gong
 2016-05-26 23:49:20 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Rohith Sharma K S
0a544f8a3e YARN-5005. TestRMWebServices#testDumpingSchedulerLogs fails randomly. Contributed by Bibin A Chundatt. 2016-05-27 10:44:35 +05:30
Eric Payne
1b0cfc84e7 YARN-5116. Failed to execute "yarn application". Contributed by Jun Gong 2016-05-26 23:43:08 +00:00
Arun Suresh
5b41b288d0 YARN-5162. Fix Exceptions thrown during in registerAM call when Distributed Scheduling is Enabled (Hitesh Sharma via asuresh) 2016-05-26 14:56:37 -07:00
Karthik Kambatla
04ded558b0 YARN-5035. FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on. (kasha) 2016-05-26 14:41:07 -07:00
Karthik Kambatla
4f513a4a8e YARN-4866. FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy. (Yufei Gu via kasha) 2016-05-25 22:13:27 -07:00
Carlo Curino
013532a95e YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino) 2016-05-25 16:55:49 -07:00
Jason Lowe
1ba31fe9e9 YARN-4459. container-executor should only kill process groups. Contributed by Jun Gong 2016-05-25 21:24:55 +00:00
Karthik Kambatla
77d5ce9069 YARN-3344. Fix warning - procfs stat file is not in the expected format. (Ravindra Kumar Naik and Akira Ajisaka via kasha) 2016-05-25 10:50:54 -07:00
Robert Kanter
e07519b8db YARN-4766. NM should not aggregate logs older than the retention policy (haibochen via rkanter) 2016-05-25 10:25:44 -07:00
Varun Vasudev
9a31e5dfef YARN-5115. Avoid setting CONTENT-DISPOSITION header in the container-logs web-service. Contributed by Xuan Gong. 2016-05-25 19:01:12 +05:30
Varun Vasudev
c380a22031 YARN-857. Localization failures should be available in container diagnostics. Contributed by Vinod Kumar Vavilapalli. 2016-05-25 19:01:12 +05:30
Junping Du
48c931331c YARN-5131. Distributed shell AM fails when extra container arrives during finishing. Contributed by Wangda Tan. 2016-05-25 06:18:11 -07:00
Rohith Sharma K S
28bd63e92b YARN-5024. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers random failure. Contributed by Bibin A Chundatt 2016-05-25 10:15:50 +05:30
Naganarasimha
edd716e99c YARN-5114. Add additional tests in TestRMWebServicesApps and rectify testInvalidAppAttempts failure in 2.8. Contributed by Bibin A Chundatt 2016-05-25 06:11:38 +08:00
Karthik Kambatla
f979d779e1 YARN-4878. Expose scheduling policy and max running apps over JMX for Yarn queues. (Yufei Gu via kasha) 2016-05-24 10:54:11 -07:00
Naganarasimha
b4078bd17b YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt 2016-05-24 08:06:53 +08:00
Karthik Kambatla
6d043aa4cf YARN-4979. FSAppAttempt demand calculation considers demands at multiple locality levels different. (Zhihai Xu via kasha) 2016-05-23 14:29:28 -07:00
Jason Lowe
ac954486c5 YARN-5055. max apps per user can be larger than max per queue. Contributed by Eric Badger 2016-05-23 15:54:42 +00:00
Jason Lowe
d1df0266cf YARN-5103. With NM recovery enabled, restarting NM multiple times results in AM restart. Contributed by Junping Du 2016-05-23 15:15:27 +00:00
Junping Du
6161d9ba52 YARN-5112. Excessive log warnings for directory permission issue on NM recovery. Contributed by Jian He. 
(cherry picked from commit 867cd2f5a2e5966d6a7c5f5accb2fce78f9e7778)
 2016-05-22 16:04:02 -07:00
Jian He
d364ceac85 YARN-5020. Fix Documentation for Yarn Capacity Scheduler on Resource Calculator. Contributed by Takashi Ohnishi 2016-05-20 11:35:26 -07:00
Varun Vasudev
0287c49107 YARN-5016. Add support for a minimum retry interval for container retries. Contributed by Jun Gong. 2016-05-20 12:28:14 +05:30
Junping Du
22fcd819f0 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 
(cherry picked from commit 2703ec68712279494d67b0d76b7ac10e7a1628be)
 2016-05-19 14:15:21 -07:00
Jian He
feb90ffcca YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong 2016-05-19 13:01:36 -07:00
Junping Du
141873ca7d YARN-5100. The YarnApplicationState is always running in ATS even application is finished. Contributed by Xuan Gong. 2016-05-19 09:13:29 -07:00
Steve Loughran
22ff9e68d1 HADOOP-12767. Update apache httpclient version to 4.5.2; httpcore to 4.4.4. Artem Aliev via stevel. 2016-05-19 14:48:18 +01:00
Akira Ajisaka
d4274c64bc YARN-5107. TestContainerMetrics fails. (aajisaka) 2016-05-19 19:53:59 +09:00
Arun Suresh
1597630681 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh) 2016-05-18 18:46:00 -07:00
Naganarasimha
f04c81c9ce YARN-4925. ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression. Contributed by Bibin A Chundatt 2016-05-19 01:30:48 +08:00
Varun Vasudev
ef1757790d YARN-4913. Yarn logs should take a -out option to write to a directory. Contributed by Xuan Gong. 2016-05-18 22:44:33 +05:30
Arun Suresh
8a9ecb7584 YARN-5090. Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster. (asuresh) 2016-05-17 19:01:29 -07:00
Jian He
0c6726e20d Addendum patch for YARN-4832. Contributed by Junping Du 2016-05-17 14:50:30 -07:00
Jian He
fa3bc3405d YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 12:52:19 -07:00
Arun Suresh
ccc93e7812 YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh) 2016-05-16 23:22:01 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Eric Payne
1217c8f6b4 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger. 2016-05-16 20:28:04 +00:00
Jason Lowe
81effb7dcd YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du 2016-05-16 15:40:23 +00:00
Arun Suresh
f45bc5a83e YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh) 2016-05-15 17:54:34 -07:00
Xuan
7be53b65f9 YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or 
RUNNING AM. Contributed by Xuan Gong
 2016-05-14 12:19:23 -07:00
Arun Suresh
f0ac18d001 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh) 2016-05-13 13:38:36 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Jason Lowe
013000fbc2 YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger 2016-05-12 20:28:36 +00:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Sangjin Lee
0bbe01f8d5 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale) 2016-05-12 10:11:23 -07:00
Rohith Sharma K S
b7ac85259c YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-12 15:17:49 +05:30
Arun Suresh
d464f4d1c4 YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh) 2016-05-11 19:10:17 -07:00
Karthik Kambatla
4b4e4c6ba8 YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha) 2016-05-11 17:36:21 -07:00
Junping Du
39f2bac38b YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong. 2016-05-11 09:28:35 -07:00
Arun Suresh
d971bf252f YARN-5073. Refactor startContainerInternal() in ContainerManager to remove unused parameter. (Konstantinos Karanasos via asuresh) 2016-05-10 22:31:38 -07:00
Vinod Kumar Vavilapalli
87f5e35133 YARN-4842. Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs. Contributed by Ram Venkatesh and Xuan Gong. 2016-05-09 22:41:18 -07:00
Naganarasimha
2750fb900f YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt 2016-05-08 22:49:25 +05:30
Yongjun Zhang
47c41e7ac7 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang) 2016-05-06 21:50:09 -07:00
Jason Lowe
b2ed6ae731 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena 2016-05-06 22:59:39 +00:00
Wangda Tan
23248f63aa getApplicationReport call may raise NPE for removed queues. (Jian He via wangda) 2016-05-06 15:30:45 -07:00
Jian He
bb62e05925 YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan 2016-05-05 12:56:21 -07:00
Jason Lowe
d0da13229c YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 2016-05-05 14:07:54 +00:00
Varun Vasudev
72b047715c YARN-4595. Add support for configurable read-only mounts when launching Docker containers. Contributed by Billie Rinaldi. 2016-05-05 13:01:54 +05:30
Vinod Kumar Vavilapalli
9e37fe3b7a YARN-4905. Improved "yarn logs" command-line to optionally show log metadata also. Contributed by Xuan Gong. 2016-05-04 14:16:03 -07:00
Wangda Tan
7bd418e48c YARN-4984. LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak. (Junping Du via wangda) 2016-05-04 11:38:55 -07:00
Junping Du
e61d431275 YARN-4920. ATS/NM should support a link to dowload/get the logs in text format. Contributed by Xuan Gong. 2016-05-04 10:35:49 -07:00
Rohith Sharma K S
75e0450593 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt 2016-05-04 09:58:26 +05:30
Jason Lowe
ed54f5f1ff YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts 2016-05-03 20:03:41 +00:00
Jason Lowe
c6b4839168 YARN-4834. ProcfsBasedProcessTree doesn't track daemonized processes. Contributed by Nathan Roberts 2016-05-03 17:27:28 +00:00
Junping Du
06413da72e YARN-4851. Metric improvements for ATS v1.5 storage components. Li Lu via junping_du. 2016-05-03 04:16:21 -07:00
Robert Kanter
9d4d30243b Remove parent's env vars from child processes 2016-04-29 09:25:51 -07:00
Varun Vasudev
0f25a1bb52 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong. 2016-04-29 16:09:07 +05:30
Jian He
4a8508501b YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe 2016-04-28 21:54:11 -07:00
Jian He
dd80042c42 YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:27:25 -07:00
Li Lu
f16722d2ef YARN-4956. findbug issue on LevelDBCacheTimelineStore. (Zhiyuan Yang via gtcarrera9) 2016-04-27 10:57:10 -07:00
Karthik Kambatla
185c3d4de1 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha) 2016-04-27 09:43:23 -07:00
Varun Vasudev
66b07d8374 YARN-4966. Improve yarn logs to fetch container logs without specifying nodeId. Contributed by Xuan Gong. 2016-04-27 15:56:36 +05:30
Jian He
4beff01354 YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode 2016-04-26 21:00:17 -07:00
Arun Suresh
341888a0aa YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh) 2016-04-26 20:12:12 -07:00
Karthik Kambatla
1a3f1482e2 YARN-4795. ContainerMetrics drops records. (Daniel Templeton via kasha) 2016-04-26 06:15:36 -07:00
Karthik Kambatla
4b1dcbbe0c YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled 2016-04-26 05:10:09 -07:00
Arun Suresh
a5fed8bbfe YARN-4991. Fix ContainerRequest Constructor to set nodelabelExpression correctly. (Bibin A Chundatt via asuresh) 2016-04-25 23:26:21 -07:00
Arun Suresh
c282a08f38 YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh) 
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
 2016-04-24 22:38:33 -07:00
Arun Suresh
b2a654c5ee YARN-4335. Allow ResourceRequests to specify ExecutionType of a request ask (kkaranasos via asuresh) 
(cherry picked from commit 8ffabfdf4fec104cdaa68133385cfe9cf88b95a8)
 2016-04-22 23:20:54 -07:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Wangda Tan
7cb3a3da96 YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda) 2016-04-22 11:40:32 -07:00
Eric Payne
3dce486d88 YARN-4556. TestFifoScheduler.testResourceOverCommit fails. Contributed by Akihiro Suda 2016-04-21 21:16:47 +00:00
Li Lu
7c6339f66a YARN-4968. A couple of AM retry unit tests need to wait SchedulerApplicationAttempt stopped. (Wangda Tan via gtcarrera9) 2016-04-21 13:25:33 -07:00
Chris Douglas
95a5046607 YARN-4976. Missing NullPointer check in ContainerLaunchContextPBImpl. Contributed by Giovanni Matteo Fumarola 2016-04-21 10:45:38 -07:00
Karthik Kambatla
170c4fd4cd YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha) 2016-04-20 23:58:12 -07:00
Karthik Kambatla
6e297836d6 YARN-4935. TestYarnClient#testSubmitIncorrectQueue fails when FairScheduler is the default. (Yufei Gu via kasha) 2016-04-20 23:40:46 -07:00
Wangda Tan
33fd95a99c YARN-4890. Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate. (Sunil G via wangda) 2016-04-20 17:37:38 -07:00
Karthik Kambatla
c8172f5f14 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha) 2016-04-20 09:55:50 -07:00
Vinod Kumar Vavilapalli
477003730e Fixed TimelineClient to retry SocketTimeoutException too. Contributed by Xuan Gong. 2016-04-18 11:47:06 -07:00
Wangda Tan
fdc46bfb37 YARN-4934. Reserved Resource for QueueMetrics needs to be handled correctly in few cases. (Sunil G via wangda) 2016-04-16 22:47:41 -07:00
Xuan
e6c0742012 YARN-4965. Distributed shell AM failed due to ClientHandlerException thrown by jersey. Contributed by Junping Du 2016-04-16 19:39:18 -07:00
Arun Suresh
cab9cbaa0a YARN-4468. Document the general ReservationSystem functionality, and the REST API. (subru and carlo via asuresh) 2016-04-15 16:58:49 -07:00