big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,669 Commits 2 Branches 2 Tags 264 MiB
29caf6d7df
Commit Graph

650 Commits

Author SHA1 Message Date
Sangjin Lee
39cce4e629 YARN-3816. [Aggregation] App-level aggregation and accumulation for YARN system metrics (Li Lu via sjlee) 2016-07-10 08:45:56 -07:00
Sangjin Lee
84c35ac6c4 YARN-4711. NM is going down with NPE's due to single thread processing of events by Timeline client (Naganarasimha G R via sjlee) 2016-07-10 08:45:55 -07:00
Varun Saxena
6f6cc647d6 YARN-4712. CPU Usage Metric is not captured properly in YARN-2928. (Naganarasimha G R via Varun Saxena) 2016-07-10 08:45:54 -07:00
Sangjin Lee
0d02ab8729 YARN-3367. Replace starting a separate thread for post entity with event loop in TimelineClient (Naganarasimha G R via sjlee) 2016-07-10 08:45:53 -07:00
Sangjin Lee
960af7d471 YARN-4409. Fix javadoc and checkstyle issues in timelineservice code (Varun Saxena via sjlee) 2016-07-10 08:45:53 -07:00
Naganarasimha
6934b05c71 YARN-4238. createdTime and modifiedTime is not reported while publishing entities to ATSv2. (Varun Saxena via Naganarasimha G R) 2016-07-10 08:45:52 -07:00
Li Lu
89e5c44f9e YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has no 
impact when it's turned off. Contributed by Sangjin Lee.
 2016-07-10 08:45:48 -07:00
Sangjin Lee
8d9476ec5f YARN-4058. Miscellaneous issues in NodeManager project (Naganarasimha G R via sjlee) 2016-07-10 08:45:43 -07:00
Junping Du
477a30f536 YARN-3045. Implement NM writing container lifecycle events to Timeline Service v2. Contributed by Naganarasimha G R. 2016-07-10 08:45:42 -07:00
Sangjin Lee
22e7ae5771 YARN-3792. Test case failures in TestDistributedShell and some issue fixes related to ATSV2 (Naganarasimha G R via sjlee) 
(cherry picked from commit 84f37f1c7eefec6d139cbf091c50d6c06f734323)
 2016-07-10 08:45:38 -07:00
Zhijie Shen
f3c661e8dd YARN-3044. Made RM write app, attempt and optional container lifecycle events to timeline service v2. Contributed by Naganarasimha G R. 2016-07-10 08:45:37 -07:00
Junping Du
d275677e24 YARN-3634. TestMRTimelineEventHandling and TestApplication are broken. Contributed by Sangjin Lee. 
(cherry picked from commit b059dd4882fd759e4762cc11c019be4b68fb74c1)
 2016-07-10 08:45:36 -07:00
Sangjin Lee
fc8485d893 YARN-3551. Consolidate data model change according to the backend implementation (Zhijie Shen via sale) 
(cherry picked from commit 557a3950bddc837469244835f5577899080115d8)
 2016-07-10 08:45:35 -07:00
Junping Du
47f35a30bb YARN-3391. Clearly define flow ID/ flow run / flow version in API and storage. Contributed by Zhijie Shen 
(cherry picked from commit 68c6232f8423e55b4d152ef3d1d66aeb2d6a555e)
 2016-07-10 08:45:33 -07:00
Zhijie Shen
5712b8f9fd YARN-3334. NM uses timeline client to publish container metrics to new timeline service. Contributed by Junping Du. 2016-07-10 08:45:33 -07:00
Junping Du
d67c9bdb4d YARN-3040. Make putEntities operation be aware of the app's context. Contributed by Zhijie Shen 2016-07-10 08:45:32 -07:00
Junping Du
2188a07e5b YARN-3333. Rename TimelineAggregator etc. to TimelineCollector. Contributed by Sangjin Lee 2016-07-10 08:45:31 -07:00
Zhijie Shen
9b56364080 YARN-3039. Implemented the app-level timeline aggregator discovery service. Contributed by Junping Du. 2016-07-10 08:45:31 -07:00
Zhijie Shen
bf54d32750 YARN-3210. Refactored timeline aggregator according to new code organization proposed in YARN-3166. Contributed by Li Lu. 
(cherry picked from commit d3ff7f06cbc66d3a23c2551e7d4c752689f46afe)
 2016-07-10 08:45:30 -07:00
Zhijie Shen
9d57c9c015 YARN-3087. Made the REST server of per-node aggregator work alone in NM daemon. Conntributed by Li Lu. 
(cherry picked from commit 41a08ad404d4278fe598d6c222b2ae0e84bae0df)
 2016-07-10 08:45:29 -07:00
Zhijie Shen
7c8abec0a8 YARN-3030. Set up TS aggregator with basic request serving structure and lifecycle. Contributed by Sangjin Lee. 
(cherry picked from commit f26941b39028ac30c77547e2be2d657bb5bf044a)
 2016-07-10 08:45:28 -07:00
Vinod Kumar Vavilapalli
4c9e1aeb94 YARN-5224. Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application. Contributed by Xuan Gong. 2016-07-06 14:10:54 -07:00
Jian He
d792a90206 YARN-5296. NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl. Contributed by Junping Du 2016-07-05 18:05:16 -07:00
Vinod Kumar Vavilapalli
ce9c006430 YARN-5214. Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks. Contributed by Junping Du. 2016-07-05 16:07:28 -07:00
Ray Chiang
0a5def155e YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang) 2016-07-01 15:53:46 -07:00
Arun Suresh
99e5dd68d0 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh) 2016-06-22 19:04:54 -07:00
Tsuyoshi Ozawa
5d58858bb6 HADOOP-9613. [JDK8] Update jersey version to latest 1.x release. 2016-06-21 08:05:32 +09:00
Sangjin Lee
5107a967fa YARN-4958. The file localization process should allow for wildcards to reduce the application footprint in the state store (Daniel Templeton via sjlee) 2016-06-20 09:56:53 -07:00
Junping Du
d0162f2040 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev. 2016-06-19 17:44:54 -07:00
Junping Du
c35fa4a0e5 YARN-5251. Yarn CLI to obtain App logs for last 'n' bytes fails. Contributed by Xuan Gong. 2016-06-17 08:24:24 -07:00
Junping Du
5dfc38ff57 YARN-5237. Fix missing log files issue in rolling log aggregation. Contributed by Xuan Gong. 2016-06-15 16:17:54 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Arun Suresh
7dae2b3bc4 YARN-5212. Run existing ContainerManager tests using QueuingContainerManagerImpl. (Konstantinos Karanasos via asuresh) 2016-06-12 10:05:37 -07:00
Vinod Kumar Vavilapalli
9378d9428f YARN-5191. Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option. (Xuan Gong via vinodkv) 2016-06-09 12:30:58 -07:00
Naganarasimha
1500a0a300 YARN-4308. ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats. Contributed by Sunil G 2016-06-09 05:41:09 +05:30
Arun Suresh
3344ba70e0 YARN-5204. Properly report status of killed/stopped queued containers. (Konstantinos Karanasos via asuresh) 2016-06-08 08:31:32 -07:00
Arun Suresh
76f0800c21 YARN-5176. More test cases for queuing of containers at the NM. (Konstantinos Karanasos via asuresh) 2016-06-07 17:16:18 -07:00
Xuan
58be55b6e0 YARN-5199. Close LogReader in in AHSWebServices#getStreamingOutput and 
FileInputStream in NMWebServices#getLogs. Contributed by Xuan Gong
 2016-06-07 16:07:02 -07:00
Rohith Sharma K S
bddea5fe5f YARN-5118. Tests fails with localizer port bind exception. Contributed by Brahma Reddy Battula. 2016-06-07 11:20:15 +05:30
Ming Ma
4a1cedc010 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma) 2016-06-06 14:30:51 -07:00
Jian He
99cc439e29 YARN-5190. Registering/unregistering container metrics in ContainerMonitorImpl and ContainerImpl causing uncaught exception in ContainerMonitorImpl. Contributed by Junping Du 2016-06-03 11:10:42 -07:00
Arun Suresh
dc26601d8f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh) 2016-06-02 09:01:02 -07:00
Xuan
0bc05e40fa YARN-5088. Improve "yarn log" command-line to read the last K bytes for the log files. Contributed by Xuan Gong 2016-06-01 13:44:21 -07:00
Varun Vasudev
86fb58b7dc YARN-4007. Add support for different network setups when launching the docker container. Contributed by Sidharta Seethana. 2016-05-29 21:32:51 +05:30
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Arun Suresh
4fc09a897b YARN-5117. QueuingContainerManager does not start GUARANTEED Container even if Resources are available. (Konstantinos Karanasos via asuresh) 2016-05-27 17:49:15 -07:00
Arun Suresh
aa975bc781 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh) 2016-05-27 14:06:32 -07:00
Carlo Curino
013532a95e YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino) 2016-05-25 16:55:49 -07:00
Jason Lowe
1ba31fe9e9 YARN-4459. container-executor should only kill process groups. Contributed by Jun Gong 2016-05-25 21:24:55 +00:00
Robert Kanter
e07519b8db YARN-4766. NM should not aggregate logs older than the retention policy (haibochen via rkanter) 2016-05-25 10:25:44 -07:00
Varun Vasudev
9a31e5dfef YARN-5115. Avoid setting CONTENT-DISPOSITION header in the container-logs web-service. Contributed by Xuan Gong. 2016-05-25 19:01:12 +05:30
Varun Vasudev
c380a22031 YARN-857. Localization failures should be available in container diagnostics. Contributed by Vinod Kumar Vavilapalli. 2016-05-25 19:01:12 +05:30
Jason Lowe
d1df0266cf YARN-5103. With NM recovery enabled, restarting NM multiple times results in AM restart. Contributed by Junping Du 2016-05-23 15:15:27 +00:00
Junping Du
6161d9ba52 YARN-5112. Excessive log warnings for directory permission issue on NM recovery. Contributed by Jian He. 
(cherry picked from commit 867cd2f5a2e5966d6a7c5f5accb2fce78f9e7778)
 2016-05-22 16:04:02 -07:00
Varun Vasudev
0287c49107 YARN-5016. Add support for a minimum retry interval for container retries. Contributed by Jun Gong. 2016-05-20 12:28:14 +05:30
Junping Du
22fcd819f0 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 
(cherry picked from commit 2703ec68712279494d67b0d76b7ac10e7a1628be)
 2016-05-19 14:15:21 -07:00
Akira Ajisaka
d4274c64bc YARN-5107. TestContainerMetrics fails. (aajisaka) 2016-05-19 19:53:59 +09:00
Arun Suresh
1597630681 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh) 2016-05-18 18:46:00 -07:00
Jian He
fa3bc3405d YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 12:52:19 -07:00
Jason Lowe
81effb7dcd YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du 2016-05-16 15:40:23 +00:00
Arun Suresh
f45bc5a83e YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh) 2016-05-15 17:54:34 -07:00
Arun Suresh
f0ac18d001 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh) 2016-05-13 13:38:36 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Sangjin Lee
0bbe01f8d5 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale) 2016-05-12 10:11:23 -07:00
Arun Suresh
d464f4d1c4 YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh) 2016-05-11 19:10:17 -07:00
Arun Suresh
d971bf252f YARN-5073. Refactor startContainerInternal() in ContainerManager to remove unused parameter. (Konstantinos Karanasos via asuresh) 2016-05-10 22:31:38 -07:00
Varun Vasudev
72b047715c YARN-4595. Add support for configurable read-only mounts when launching Docker containers. Contributed by Billie Rinaldi. 2016-05-05 13:01:54 +05:30
Wangda Tan
7bd418e48c YARN-4984. LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak. (Junping Du via wangda) 2016-05-04 11:38:55 -07:00
Junping Du
e61d431275 YARN-4920. ATS/NM should support a link to dowload/get the logs in text format. Contributed by Xuan Gong. 2016-05-04 10:35:49 -07:00
Robert Kanter
9d4d30243b Remove parent's env vars from child processes 2016-04-29 09:25:51 -07:00
Varun Vasudev
0f25a1bb52 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong. 2016-04-29 16:09:07 +05:30
Jian He
4a8508501b YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe 2016-04-28 21:54:11 -07:00
Arun Suresh
341888a0aa YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh) 2016-04-26 20:12:12 -07:00
Karthik Kambatla
1a3f1482e2 YARN-4795. ContainerMetrics drops records. (Daniel Templeton via kasha) 2016-04-26 06:15:36 -07:00
Arun Suresh
c282a08f38 YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh) 
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
 2016-04-24 22:38:33 -07:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Karthik Kambatla
c8172f5f14 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha) 2016-04-20 09:55:50 -07:00
Jason Lowe
3150ae8108 YARN-4924. NM recovery race can lead to container not cleaned up. Contributed by sandflee 2016-04-14 19:17:14 +00:00
Robert Kanter
0d9194df00 YARN-4541. Change log message in LocalizedResource#handle() to DEBUG (rchiang via rkanter) 2016-04-13 17:45:18 -07:00
Naganarasimha
437e9d6475 YARN-4810. NM applicationpage cause internal error 500. Contributed by Bibin A Chundatt. 2016-04-12 17:59:46 +05:30
Vinod Kumar Vavilapalli
44bbc50d91 YARN-4168. Fixed a failing test TestLogAggregationService.testLocalFileDeletionOnDiskFull. Contributed by Takashi Ohnishi. 2016-04-11 12:11:14 -07:00
Akira Ajisaka
1ff27f9d12 YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-04-11 14:55:03 +09:00
Karthik Kambatla
e82f961a39 YARN-4756. Unnecessary wait in Node Status Updater during reboot. (Eric Badger via kasha) 2016-04-07 17:05:29 -07:00
Varun Vasudev
b41e65e5bc YARN-4906. Capture container start/finish time in container metrics. Contributed by Jian He. 2016-04-06 13:41:33 +05:30
Junping Du
0005816743 YARN-4916. TestNMProxy.tesNMProxyRPCRetry fails. Contributed by Tibor Kiss. 2016-04-05 09:01:08 -07:00
naganarasimha
5092c94195 YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt 2016-04-04 16:25:03 +05:30
Jian He
0dd9bcab97 YARN-4811. Generate histograms in ContainerMetrics for actual container resource usage 2016-03-31 14:28:13 -07:00
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Jason Lowe
948b758070 YARN-4773. Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled. Contributed by Jun Gong 2016-03-28 23:00:56 +00:00
Jian He
55ae143923 YARN-4117. End to end unit test with mini YARN cluster for AMRMProxy Service. Contributed by Giovanni Matteo Fumarola 2016-03-27 20:22:12 -07:00
Eric Payne
92b7e0d413 YARN-4686. MiniYARNCluster.start() returns before cluster is completely started. Contributed by Eric Badger. 2016-03-18 16:12:47 +00:00
Robert Kanter
22ca176dfe TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup (templedf via rkanter) 2016-03-15 10:05:10 -07:00
Vinod Kumar Vavilapalli
b2661765a5 YARN-4762. Fixed CgroupHandler's creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled. (Sidharta Seethana via vinodkv) 2016-03-07 11:08:17 -08:00
Jason Lowe
059caf9989 YARN-4744. Too many signal to container failure in case of LCE. Contributed by Sidharta Seethana 2016-03-07 15:40:01 +00:00
Varun Vasudev
8ed2e060e8 YARN-4245. Generalize config file handling in container-executor. Contributed by Sidharta Seethana. 2016-03-07 16:18:35 +05:30
Varun Vasudev
e51a8c1056 YARN-4737. Add CSRF filter support in YARN. Contributed by Jonathan Maron. 2016-03-07 15:26:44 +05:30
Haohui Mai
0fa54d45b1 HADOOP-12813. Migrate TestRPC and related codes to rebase on ProtobufRpcEngine. Contributed by Kai Zheng. 2016-02-29 11:41:00 -08:00
Jason Lowe
c58a6d53c5 YARN-4731. container-executor should not follow symlinks in recursive_unlink_children. Contributed by Colin Patrick McCabe 2016-02-29 15:24:35 +00:00