Hadoop YARN Change Log Trunk - Unreleased INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS YARN-84. Use Builder to build RPC server. (Brandon Li via suresh) OPTIMIZATIONS BUG FIXES YARN-524 TestYarnVersionInfo failing if generated properties doesn't include an SVN URL. (stevel) Release 2.3.0 - UNRELEASED INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS BUG FIXES Release 2.1.1-beta - UNRELEASED INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS BUG FIXES Release 2.1.0-beta - 2013-07-02 INCOMPATIBLE CHANGES YARN-396. Rationalize AllocateResponse in RM Scheduler API. (Zhijie Shen via hitesh) YARN-439. Flatten NodeHeartbeatResponse. (Xuan Gong via sseth) YARN-440. Flatten RegisterNodeManagerResponse. (Xuan Gong via sseth) YARN-536. Removed the unused objects ContainerStatus and ContainerStatus from Container which also don't belong to the container. (Xuan Gong via vinodkv) YARN-486. Changed NM's startContainer API to accept Container record given by RM as a direct parameter instead of as part of the ContainerLaunchContext record. (Xuan Gong via vinodkv) YARN-444. Moved special container exit codes from YarnConfiguration to API where they belong. (Sandy Ryza via vinodkv) YARN-441. Removed unused utility methods for collections from two API records. (Xuan Gong via vinodkv) YARN-561. Modified NodeManager to set key information into the environment of every container that it launches. (Xuan Gong via vinodkv) YARN-579. Stop setting the Application Token in the AppMaster env, in favour of the copy present in the container token field. (Vinod Kumar Vavilapalli via sseth) YARN-629. Make YarnRemoteException not be rooted at IOException. (Xuan Gong via vinodkv) YARN-633. Changed RMAdminProtocol api to throw IOException and YarnRemoteException. (Xuan Gong via vinodkv) YARN-632. Changed ContainerManager api to throw IOException and YarnRemoteException. (Xuan Gong via vinodkv) YARN-631. Changed ClientRMProtocol api to throw IOException and YarnRemoteException. (Xuan Gong via vinodkv) YARN-630. Changed AMRMProtocol api to throw IOException and YarnRemoteException. (Xuan Gong via vinodkv) YARN-615. Rename ContainerLaunchContext.containerTokens to tokens. (Vinod Kumar Vavilapalli via sseth) YARN-571. Remove user from ContainerLaunchContext. (Omkar Vinit Joshi via vinodkv) YARN-716. Making ApplicationID immutable. (Siddharth Seth via vinodkv) YARN-684. ContainerManager.startContainer should use ContainerTokenIdentifier instead of the entire Container. (Vinod Kumar Vavilapalli via sseth) YARN-735. Make ApplicationAttemptId, ContaienrId and NodeId immutable. (Jian He via sseth) YARN-749. Rename ResourceRequest.(get,set)HostName to ResourceRequest.(get,set)ResourceName. (acmurthy) YARN-720. container-log4j.properties should not refer to mapreduce property names. (Zhijie Shen via sseth) YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. (Jian He via vinodkv) YARN-635. Renamed YarnRemoteException to YarnException. (Siddharth Seth via vinodkv) YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. (Bikas Saha via vinodkv) YARN-753. Added individual factory methods for all api protocol records and converted the records to be abstract classes. (Jian He via vinodkv) YARN-724. Moved ProtoBase from api.records to api.records.impl.pb. (Jian He via vinodkv) YARN-759. Create Command enum in AllocateResponse (bikas) YARN-777. Removed unreferenced objects from .proto files. (Jian He via vinodkv) YARN-642. Removed health parameter from ResourceManager /nodes web-service and cleaned the behaviour of the status parameter. (Sandy Ryza vid vinodkv) YARN-530. Defined Service model strictly, implemented AbstractService for robust subclassing and migrated yarn-common services. (Steve Loughran via vinodkv) YARN-746. Renamed Service.register() and Service.unregister() to registerServiceListener() & unregisterServiceListener() respectively. (Steve Loughran via vinodkv) YARN-792. Moved NodeHealthStatus from yarn.api.record to yarn.server.api.record. (Jian He via vinodkv) YARN-806. Moved ContainerExitStatus from yarn.api to yarn.api.records. (Jian He via vinodkv) YARN-821. Renamed setFinishApplicationStatus to setFinalApplicationStatus in FinishApplicationMasterRequest for consistency. (Jian He via vinodkv) YARN-787. Removed minimum resource from RegisterApplicationMasterResponse. (tucu via acmurthy) YARN-829. Renamed RMTokenSelector to be RMDelegationTokenSelector. (Zhijie Shen via vinodkv) YARN-828. Removed the unsed YarnVersionAnnotation. (Zhijie Shen via vinodkv) YARN-823. Moved RMAdmin from yarn.client to yarn.client.cli and renamed it to be RMAdminCLI. (Jian He via vinodkv) YARN-387. Renamed YARN protocols for consistency. ClientRMProtocol -> ApplicationClientProtocol AMRMProtocol -> ApplicationMasterProtocol ContainerManager -> ContainerManagementProtocol (vinodkv via acmurthy) YARN-831. Removed minimum resource from GetNewApplicationResponse as a follow-up to YARN-787. (Jian He via acmurthy) YARN-824. Added static factory methods to hadoop-yarn-client interfaces. (Jian He via acmurthy) YARN-826. Moved Clock and SystemClock into yarn.util package. (Zhijie Shen via vinodkv) YARN-837. Moved yarn.ClusterInfo into MapReduce project as it doesn't belong to YARN. (Zhijie Shen via vinodkv) YARN-822. Renamed ApplicationToken to be AMRMToken, and similarly the corresponding TokenSelector and SecretManager. (Omkar Vinit Joshi via vinodkv) YARN-610. ClientToken is no longer set in the environment of the Containers. (Omkar Vinit Joshi via vinodkv) YARN-834. Fixed annotations for yarn-client module, reorganized packages and clearly differentiated *Async apis. (Arun C Murthy and Zhijie Shen via vinodkv) YARN-840. Moved ProtoUtils to yarn.api.records.pb.impl. (Jian He via acmurthy) YARN-841. Move Auxiliary service to yarn-api, annotate and document it. (vinodkv) YARN-850. Rename getClusterAvailableResources to getAvailableResources in AMRMClients (Jian He via bikas) YARN-694. Starting to use NMTokens to authenticate all communication with NodeManagers. (Omkar Vinit Joshi via vinodkv) YARN-553. Replaced YarnClient.getNewApplication with YarnClient.createApplication which provides a directly usable ApplicationSubmissionContext to simplify the api. (Karthik Kambatla via acmurthy) YARN-851. Share NMTokens using NMTokenCache (api-based) between AMRMClient and NMClient instead of memory based approach which is used currently. (Omkar Vinit Joshi via vinodkv) YARN-869. Move ResourceManagerAdministrationProtocol out of main YARN api. (vinodkv via acmurthy) YARN-791. Changed RM APIs and web-services related to nodes to ensure that both are consistent with each other. (Sandy Ryza via vinodkv) YARN-727. ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter. (Xuan Gong via hitesh) YARN-701. Use application tokens irrespective of secure or non-secure mode. (vinodkv via acmurthy) YARN-918. Remove ApplicationAttemptId from RegisterApplicationMasterRequestProto. (vinodkv via acmurthy) YARN-926. Modified ContainerManagerProtcol APIs to take in requests for multiple containers. (Jian He via vinodkv) NEW FEATURES YARN-482. FS: Extend SchedulingMode to intermediate queues. (kkambatl via tucu) YARN-45. Add protocol for schedulers to request containers back from ApplicationMasters. (Carlo Curino, cdouglas) YARN-563. Add the concept of an application-type for each application. (Mayank Bansal via vinodkv) HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows Azure environments. (See breakdown of tasks below for subtasks and contributors) YARN-422. Add a NM Client library to help application-writers. (Zhijie Shen via vinodkv) YARN-392. Make it possible to specify hard locality constraints in resource requests. (sandyr via tucu) YARN-326. Add multi-resource scheduling to the fair scheduler. (sandyr via tucu) YARN-398. Make it possible to specify hard locality constraints in resource requests for CapacityScheduler. (acmurthy) YARN-781. Exposing LOGDIR in all containers' environment which should be used by containers for logging purposes. (Jian He via vinodkv) IMPROVEMENTS YARN-365. Change NM heartbeat handling to not generate a scheduler event on each heartbeat. (Xuan Gong via sseth) YARN-380. Fix yarn node -status output to be better readable. (Omkar Vinit Joshi via vinodkv) YARN-410. Fixed RM UI so that the new lines diagnostics for a failed app on the per-application page are translated to html line breaks. (Omkar Vinit Joshi via vinodkv) YARN-198. Added a link to RM pages from the NodeManager web app. (Jian He via vinodkv) YARN-237. Refreshing the RM page forgets how many rows I had in my Datatables (jian he via bobby) YARN-481. Add AM Host and RPC Port to ApplicationCLI Status Output (Chris Riccomini via bikas) YARN-297. Improve hashCode implementations for PB records. (Xuan Gong via hitesh) YARN-417. Create AMRMClient wrapper that provides asynchronous callbacks. (Sandy Ryza via bikas) YARN-497. Yarn unmanaged-am launcher jar does not define a main class in its manifest (Hitesh Shah via bikas) YARN-469. Make scheduling mode in FS pluggable. (kkambatl via tucu) YARN-450. Define value for * in the scheduling protocol (Zhijie Shen via bikas) YARN-475. Remove a unused constant in the public API - ApplicationConstants.AM_APP_ATTEMPT_ID_ENV. (Hitesh Shah via vinodkv) YARN-309. Changed NodeManager to obtain heart-beat interval from the ResourceManager. (Xuan Gong via vinodkv) YARN-447. Move ApplicationComparator in CapacityScheduler to use comparator in ApplicationId. (Nemon Lou via vinodkv) YARN-381. Improve fair scheduler docs. (Sandy Ryza via tomwhite) YARN-458. YARN daemon addresses must be placed in many different configs. (sandyr via tucu) YARN-193. Scheduler.normalizeRequest does not account for allocation requests that exceed maximumAllocation limits (Zhijie Shen via bikas) YARN-479. NM retry behavior for connection to RM should be similar for lost heartbeats (Jian He via bikas) YARN-495. Changed NM reboot behaviour to be a simple resync - kill all containers and re-register with RM. (Jian He via vinodkv) YARN-514. Delayed store operations should not result in RM unavailability for app submission (Zhijie Shen via bikas) YARN-586. Fixed a typo in ApplicationSubmissionContext#setApplicationId. (Zhijie Shen via vinodkv) YARN-542. Changed the default global AM max-attempts value to be not one. (Zhijie Shen via vinodkv) YARN-583. Moved application level local resources to be localized under the filecache sub-directory under application directory. (Omkar Vinit Joshi via vinodkv) YARN-581. Added a test to verify that app delegation tokens are restored after RM restart. (Jian He via vinodkv) YARN-577. Add application-progress also to ApplicationReport. (Hitesh Shah via vinodkv) YARN-595. Refactor fair scheduler to use common Resources. (Sandy Ryza via tomwhite) YARN-562. Modified NM to reject any containers allocated by a previous ResourceManager. (Jian He via vinodkv) YARN-591. Moved RM recovery related records out of public API as they do not belong there. (vinodkv) YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager to separate out various validation checks depending on whether they rely on RM configuration or not. (Zhijie Shen via vinodkv) YARN-618. Modified RM_INVALID_IDENTIFIER to be -1 instead of zero. (Jian He via vinodkv) YARN-625. Move the utility method unwrapAndThrowException from YarnRemoteExceptionPBImpl to RPCUtil. (Siddharth Seth via vinodkv) YARN-645. Moved RMDelegationTokenSecretManager from yarn-server-common to yarn-server-resourcemanager where it really belongs. (Jian He via vinodkv) YARN-651. Changed PBClientImpls of ContainerManager and RMAdmin to throw IOExceptions also. (Xuan Gong via vinodkv) YARN-582. Changed ResourceManager to recover Application token and client tokens for app attempt so that RM can be restarted while preserving current applications. (Jian He via vinodkv) YARN-568. Add support for work preserving preemption to the FairScheduler. (Carlo Curino and Sandy Ryza via cdouglas) YARN-598. Add virtual cores to queue metrics. (sandyr via tucu) YARN-634. Modified YarnRemoteException to be not backed by PB and introduced a separate SerializedException record. (Siddharth Seth via vinodkv) YARN-663. Changed ResourceTracker API and LocalizationProtocol API to throw YarnRemoteException and IOException. (Xuan Gong via vinodkv) YARN-590. Added an optional mesage to be returned by ResourceMaanger when RM asks an RM to shutdown/resync etc so that NMs can log this message locally for better debuggability. (Mayank Bansal via vinodkv) YARN-617. Made ContainerTokens to be used for validation at NodeManager also in unsecure mode to prevent AMs from faking resource requirements in unsecure mode. (Omkar Vinit Joshi via vinodkv) YARN-708. Moved RecordFactory classes to hadoop-yarn-api, and put some miscellaneous fixes to the interfaces. (Siddharth Seth via vinodkv) YARN-711. Copied BuilderUtil methods in individual API records as BuilderUtils is going to be dismantled. (Jian He via vinodkv) YARN-714. Added NMTokens to be sent to AMs as part of heart-beat response. (Omkar Vinit Joshi via vinodkv) YARN-638. Modified ResourceManager to restore RMDelegationTokens after restarting. (Jian He via vinodkv) YARN-660. Improve AMRMClient with matching requests (bikas) YARN-717. Put object creation factories for Token in the class itself and remove useless derivations for specific tokens. (Jian He via vinodkv) YARN-756. Move Preemption* records to yarn.api where they really belong. (Jian He via vinodkv) YARN-750. Allow for black-listing resources in YARN API and Impl in CS (acmurthy via bikas) YARN-877. Support resource blacklisting for FifoScheduler. (Junping Du via llu) YARN-686. Flatten NodeReport. (sandyr via tucu) YARN-737. Throw some specific exceptions directly instead of wrapping them in YarnException. (Jian He via sseth) YARN-731. RPCUtil.unwrapAndThrowException should unwrap remote RuntimeExceptions. (Zhijie Shen via sseth) YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu) YARN-773. Moved YarnRuntimeException from package api.yarn to api.yarn.exceptions. (Jian He via vinodkv) YARN-692. Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat. (Omkar Vinit Joshi via vinodkv) YARN-782. vcores-pcores ratio functions differently from vmem-pmem ratio in misleading way. (sandyr via tucu) YARN-803. factor out scheduler config validation from the ResourceManager to each scheduler implementation. (tucu) YARN-789. Enable zero capabilities resource requests in fair scheduler. (tucu) YARN-639. Modified Distributed Shell application to start using the new NMClient library. (Zhijie Shen via vinodkv) YARN-693. Modified RM to send NMTokens on allocate call so that AMs can then use them for authentication with NMs. (Omkar Vinit Joshi via vinodkv) YARN-752. In AMRMClient, automatically add corresponding rack requests for requested nodes. (sandyr via tucu) YARN-825. Fixed javadoc and annotations for yarn-common module. (vinodkv) YARN-833. Moved Graph and VisualizeStateMachine into yarn.state package. (Zhijie Shen via vinodkv) YARN-805. Fix javadoc and annotations on classes in the yarn-api package. (Jian He via sseth) YARN-846. Move pb Impl classes from yarn-api to yarn-common. (Jian He via vinodkv) YARN-827. Need to make Resource arithmetic methods accessible (Jian He via bikas) YARN-866. Add test for class ResourceWeights. (ywskycn via tucu) YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu) YARN-569. Add support for requesting and enforcing preemption requests via a capacity monitor. (Carlo Curino, cdouglas) YARN-521. Augment AM - RM client module to be able to request containers only at specific locations (Sandy Ryza via bikas) YARN-513. Create common proxy client for communicating with RM. (Xuan Gong & Jian He via bikas) YARN-927. Change ContainerRequest to not have more than 1 container count and remove StoreContainerRequest (bikas) YARN-922. Change FileSystemRMStateStore to use directories (Jian He via bikas) YARN-865. RM webservices can't query based on application Types. (Xuan Gong via hitesh) YARN-912. Move client facing exceptions to yarn-api module. (Mayank Bansal via vinodkv) OPTIMIZATIONS YARN-512. Log aggregation root directory check is more expensive than it needs to be. (Maysam Yabandeh via jlowe) YARN-719. Move RMIdentifier from Container to ContainerTokenIdentifier. (Vinod Kumar Vavilapalli via sseth) BUG FIXES YARN-383. AMRMClientImpl should handle null rmClient in stop() (Hitesh Shah via sseth) YARN-385. Add missing fields - location and #containers to ResourceRequestPBImpl's toString(). (Sandy Ryza via sseth) YARN-377. Use the new StringUtils methods added by HADOOP-9252 and fix TestContainersMonitor. (Chris Nauroth via szetszwo) YARN-391. Formatting fixes for LCEResourceHandler classes. (Steve Loughran via sseth) YARN-390. ApplicationCLI and NodeCLI hard-coded platform-specific line separator causes test failures on Windows. (Chris Nauroth via suresh) YARN-406. Fix TestRackResolver to function in networks where "host1" resolves to a valid host. (Hitesh Shah via sseth) YARN-376. Fixes a bug which would prevent the NM knowing about completed containers and applications. (Jason Lowe via sseth) YARN-196. Nodemanager should be more robust in handling connection failure to ResourceManager when a cluster is started (Xuan Gong via hitesh) YARN-485. TestProcfsProcessTree#testProcessTree() doesn't wait long enough for the process to die. (kkambatl via tucu) YARN-71. Fix the NodeManager to clean up local-dirs on restart. (Xuan Gong via sseth) YARN-378. Fix RM to make the AM max attempts/retries to be configurable per application by clients. (Zhijie Shen via vinodkv) YARN-498. Unmanaged AM launcher does not set various constants in env for an AM, also does not handle failed AMs properly. (Hitesh Shah via bikas) YARN-496. Fair scheduler configs are refreshed inconsistently in reinitialize. (Sandy Ryza via tomwhite) YARN-474. Fix CapacityScheduler to trigger application-activation when am-resource-percent configuration is refreshed. (Zhijie Shen via vinodkv) YARN-209. Fix CapacityScheduler to trigger application-activation when the cluster capacity changes. (Zhijie Shen via vinodkv) YARN-24. Nodemanager fails to start if log aggregation enabled and namenode unavailable. (sandyr via tucu) YARN-515. Node Manager not getting the master key. (Robert Joseph Evans via jlowe) YARN-382. SchedulerUtils improve way normalizeRequest sets the resource capabilities. (Zhijie Shen via bikas) YARN-467. Modify public distributed cache to localize files such that no local directory hits unix file count limits and thus prevent job failures. (Omkar Vinit Joshi via vinodkv) YARN-101. Fix NodeManager heartbeat processing to not lose track of completed containers in case of dropped heartbeats. (Xuan Gong via vinodkv) YARN-538. RM address DNS lookup can cause unnecessary slowness on every JHS page load. (sandyr via tucu) YARN-532. Change RMAdmin and Localization client protocol PB implementations to implement closeable so that they can be stopped when needed via RPC.stopProxy(). (Siddharth Seth via vinodkv) YARN-99. Modify private distributed cache to localize files such that no local directory hits unix file count limits and thus prevent job failures. (Omkar Vinit Joshi via vinodkv) YARN-112. Fixed a race condition during localization that fails containers. (Omkar Vinit Joshi via vinodkv) YARN-534. Change RM restart recovery to also account for AM max-attempts configuration after the restart. (Jian He via vinodkv) YARN-539. Addressed memory leak of LocalResource objects NM when a resource localization fails. (Omkar Vinit Joshi via vinodkv) YARN-319. Submitting a job to a fair scheduler queue for which the user does not have permission causes the client to wait forever. (shenhong via tomwhite) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. (Roger Hoover via acmurthy) YARN-500. Fixed YARN webapps to not roll-over ports when explicitly asked to use non-ephemeral ports. (Kenji Kikushima via vinodkv) YARN-518. Fair Scheduler's document link could be added to the hadoop 2.x main doc page. (sandyr via tucu) YARN-476. ProcfsBasedProcessTree info message confuses users. (sandyr via tucu) YARN-585. Fix failure in TestFairScheduler#testNotAllowSubmitApplication caused by YARN-514. (Zhijie Shen via vinodkv) YARN-547. Fixed race conditions in public and private resource localization which used to cause duplicate downloads. (Omkar Vinit Joshi via vinodkv) YARN-594. Update test and add comments in YARN-534 (Jian He via bikas) YARN-549. YarnClient.submitApplication should wait for application to be accepted by the RM (Zhijie Shen via bikas) YARN-605. Fix failing unit test in TestNMWebServices when versionInfo has parantheses like when running on a git checkout. (Hitesh Shah via vinodkv) YARN-289. Fair scheduler allows reservations that won't fit on node. (Sandy Ryza via tomwhite) YARN-576. Modified ResourceManager to reject NodeManagers that don't satisy minimum resource requirements. (Kenji Kikushima via vinodkv) YARN-646. Fix two typos in Fair Scheduler user guide. (Dapeng Sun via atm) YARN-507. Add interface visibility and stability annotations to FS interfaces/classes. (kkambatl via tucu) YARN-637. FS: maxAssign is not honored. (kkambatl via tucu) YARN-655. Fair scheduler metrics should subtract allocated memory from available memory. (sandyr via tucu) YARN-628. Fix the way YarnRemoteException is being unrolled to extract out the underlying exception. (Siddharth Seth via vinodkv) YARN-695. Remove masterContainer and status unused fields from ApplicationReportProto and fix bugs in ApplicationReportPBImpl. (Zhijie Shen via vinodkv) YARN-706. Fixed race conditions in TestFSDownload. (Zhijie Shen via vinodkv). YARN-715. Fixed unit test failures - TestDistributedShell and TestUnmanagedAMLauncher. (Vinod Kumar Vavilapalli via sseth) YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. (Omkar Vinit Joshi via vinodkv) YARN-733. Fixed TestNMClient from failing occasionally. (Zhijie Shen via vinodkv) YARN-730. Fix NMClientAsync to remove completed containers. (Zhijie Shen via acmurthy) YARN-726. Fix queue & finish time fields in web-ui for ResourceManager. (Mayank Bansal via acmurthy) YARN-757. Changed TestRMRestart to use the default scheduler to avoid test failures. (Bikas Saha via vinodkv) YARN-742. Log aggregation causes a lot of redundant setPermission calls. (jlowe via kihwal) YARN-764. blank Used Resources on Capacity Scheduler page (Nemon Lou via tgraves) YARN-761. TestNMClientAsync fails sometimes (Zhijie Shen via bikas) YARN-760. NodeManager throws AvroRuntimeException on failed start. (Niranjan Singh via jlowe) YARN-767. Initialize application metrics at RM bootup. (Jian He via acmurthy) YARN-700. TestInfoBlock fails on Windows because of line ending missmatch. (Ivan Mitic via cnauroth) YARN-117. Migrated rest of YARN to the new service model. (Steve Louhran via vinodkv) YARN-812. Set default logger for application summary logger to hadoop.root.logger. (sseth via acmurthy) YARN-848. Nodemanager does not register with RM using the fully qualified hostname. (Hitesh Shah via sseth) YARN-854. Fixing YARN bugs that are failing applications in secure environment. (Omkar Vinit Joshi via vinodkv) YARN-861. TestContainerManager is failing. (Vinod Kumar Vavilapalli via hitesh) YARN-874. Making common RPC to switch to not switch to simple when other mechanisms are enabled and thus fix YARN/MR test failures after HADOOP-9421. (Daryn Sharp and Vinod Kumar Vavilapalli via vinodkv) YARN-845. RM crash with NPE on NODE_UPDATE (Mayank Bansal via bikas) YARN-369. Handle ( or throw a proper error when receiving) status updates from application masters that have not registered (Mayank Bansal & Abhishek Kapoor via bikas) YARN-541. getAllocatedContainers() is not returning all the allocated containers (bikas) YARN-763. AMRMClientAsync should stop heartbeating after receiving shutdown from RM (Xuan Gong via bikas) YARN-654. AMRMClient: Perform sanity checks for parameters of public methods (Xuan Gong via bikas)" YARN-919. Document setting default heap sizes in yarn-env.sh (Mayank Bansal via hitesh) YARN-795. Fair scheduler queue metrics should subtract allocated vCores from available vCores. (ywskycn via tucu) YARN-799. Fix CgroupsLCEResourcesHandler to use /tasks instead of /cgroup.procs. (Chris Riccomini via acmurthy) YARN-333. Schedulers cannot control the queue-name of an application. (sandyr via tucu) YARN-368. Fixed a typo in error message in Auxiliary services. (Albert Chu via vinodkv) YARN-295. Fixed a race condition in ResourceManager RMAppAttempt state machine. (Mayank Bansal via vinodkv) YARN-523. Modified a test-case to validate container diagnostics on localization failures. (Jian He via vinodkv) YARN-661. Fixed NM to cleanup users' local directories correctly when starting up. (Omkar Vinit Joshi via vinodkv) YARN-820. Fixed an invalid state transition in NodeManager caused by failing resource localization. (Mayank Bansal via vinodkv) YARN-62. Modified NodeManagers to avoid AMs from abusing container tokens for repetitive container launches. (Omkar Vinit Joshi via vinodkv) YARN-814. Improving diagnostics when containers fail during launch due to various reasons like invalid env etc. (Jian He via vinodkv) YARN-897. Ensure child queues are ordered correctly to account for completed containers. (Djellel Eddine Difallah via acmurthy) YARN-853. Fixed CapacityScheduler's maximum-am-resource-percent to properly work beyond refreshing queues. (Devaraj K via vinodkv) YARN-873. YARNClient.getApplicationReport(unknownAppId) returns a null report (Xuan Gong via bikas) YARN-875. Application can hang if AMRMClientAsync callback thread has exception (Xuan Gong via bikas) YARN-461. Fair scheduler should not accept apps with empty string queue name. (ywskycn via tucu) YARN-968. RM admin commands don't work. (vinodkv via kihwal) YARN-688. Fixed NodeManager to properly cleanup containers when it is shut down. (Jian He via vinodkv) BREAKDOWN OF HADOOP-8562/YARN-191 SUBTASKS AND RELATED JIRAS YARN-158. Yarn creating package-info.java must not depend on sh. (Chris Nauroth via suresh) YARN-176. Some YARN tests fail to find winutils. (Chris Nauroth via suresh) YARN-207. YARN distribution build fails on Windows. (Chris Nauroth via suresh) YARN-199. Yarn cmd line scripts for windows. (Ivan Mitic via suresh) YARN-213. YARN build script would be more readable using abspath. (Chris Nauroth via suresh) YARN-233. Added support for running containers in MS Windows to YARN. (Chris Nauroth via acmurthy) YARN-234. Added support for process tree and resource calculator in MS Windows to YARN. (Chris Nauroth via acmurthy) YARN-259. Fix LocalDirsHandlerService to use Path rather than URIs. (Xuan Gong via acmurthy) YARN-316. YARN container launch may exceed maximum Windows command line length due to long classpath. (Chris Nauroth via suresh) YARN-359. Fixing commands for container signalling in Windows. (Chris Nauroth via vinodkv) YARN-506. Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) YARN-488. TestContainerManagerSecurity fails on Windows. (Chris Nauroth via hitesh) YARN-490. TestDistributedShell fails on Windows. (Chris Nauroth via hitesh) YARN-491. TestContainerLogsPage fails on Windows. (Chris Nauroth via hitesh) YARN-487. Modify path manipulation in LocalDirsHandlerService to let TestDiskFailures pass on Windows. (Chris Nauroth via vinodkv) YARN-593. container launch on Windows does not correctly populate classpath with new process's environment variables and localized resources (Chris Nauroth via bikas) YARN-493. Fixed some shell related flaws in YARN on Windows. (Chris Nauroth via vinodkv) YARN-839. TestContainerLaunch.testContainerEnvVariables fails on Windows. (Chuan Liu via cnauroth) YARN-597. TestFSDownload fails on Windows due to dependencies on tar/gzip/jar tools. (Ivan Mitic via acmurthy) YARN-852. TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows. (Chuan Liu via cnauroth) YARN-894. NodeHealthScriptRunner timeout checking is inaccurate on Windows. (Chuan Liu via cnauroth) YARN-909. Disable TestLinuxContainerExecutorWithMocks on Windows. (Chuan Liu via cnauroth) Release 2.0.5-alpha - 06/06/2013 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS BUG FIXES Release 2.0.4-alpha - 2013-04-25 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS BUG FIXES YARN-429. capacity-scheduler config missing from yarn-test artifact. (sseth via hitesh) YARN-470. Support a way to disable resource monitoring on the NodeManager. (Siddharth Seth via hitesh) Release 2.0.3-alpha - 2013-02-06 INCOMPATIBLE CHANGES NEW FEATURES YARN-145. Add a Web UI to the fair share scheduler. (Sandy Ryza via tomwhite) YARN-3. Add support for CPU isolation/monitoring of containers. (adferguson via tucu) YARN-230. RM Restart phase 1 - includes support for saving/restarting all applications on an RM bounce. (Bikas Saha via acmurthy) YARN-103. Add a yarn AM-RM client module. (Bikas Saha via sseth) YARN-286. Add a YARN ApplicationClassLoader. (tomwhite) YARN-2. Enhanced CapacityScheduler to account for CPU alongwith memory for multi-dimensional resource scheduling. (acmurthy) YARN-328. Use token request messages defined in hadoop common. (suresh) YARN-231. RM Restart - Add FS-based persistent store implementation for RMStateStore (Bikas Saha via hitesh) IMPROVEMENTS YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. (Radim Kolar via acmurthy) YARN-78. Changed UnManagedAM application to use YarnClient. (Bikas Saha via vinodkv) YARN-93. Fixed RM to propagate diagnostics from applications that have finished but failed (Jason Lowe via vinodkv). YARN-28. Fixed TestCompositeService to not depend on test-order and thus made it pass on JDK7 (Thomas Graves via vinodkv). YARN-82. Change the default local and log dirs to be based on hadoop.tmp.dir and yarn.log.dir. (Hemanth Yamijala via sseth) YARN-53. Added the missing getGroups API to ResourceManager. (Bo Wang via vinodkv) YARN-116. Add the ability to change the RM include/exclude file without a restart. (xieguiming and Harsh J via sseth) YARN-23. FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation. (kkambatl via tucu) YARN-127. Move RMAdmin tool to its correct location - the client module. (vinodkv) YARN-40. Provided support for missing YARN commands (Devaraj K and Vinod Kumar Vavilapalli via vinodkv) YARN-33. Change LocalDirsHandlerService to validate the configured local and log dirs. (Mayank Bansal via sseth) YARN-94. Modify DistributedShell to point to main-class by default, clean up the help message, and hard-code the AM class. (Hitesh Shah via vinodkv) YARN-146. Add unit tests for computing fair share in the fair scheduler. (Sandy Ryza via tomwhite) HADOOP-8911. CRLF characters in source and text files. (Raja Aluri via suresh) YARN-136. Make ClientToAMTokenSecretManager part of RMContext (Vinod Kumar Vavilapalli via sseth) YARN-183. Clean up fair scheduler code. (Sandy Ryza via tomwhite) YARN-129. Simplify classpath construction for mini YARN tests. (tomwhite) YARN-254. Update fair scheduler web UI for hierarchical queues. (sandyr via tucu) YARN-315. Using the common security token protobuf definition from hadoop common. (Suresh Srinivas via vinodkv) YARN-170. Change NodeManager stop to be reentrant. (Sandy Ryza via vinodkv) YARN-331. Fill in missing fair scheduler documentation. (sandyr via tucu) YARN-277. Use AMRMClient in DistributedShell to exemplify the approach. (Bikas Saha via hitesh) YARN-360. Allow apps to concurrently register tokens for renewal. (Daryn Sharp via sseth) OPTIMIZATIONS BUG FIXES YARN-131. Fix incorrect ACL properties in capacity scheduler documentation. (Ahmed Radwan via sseth) YARN-102. Move the apache header to the top of the file in MemStore.java. (Devaraj K via sseth) YARN-134. ClientToAMSecretManager creates keys without checking for validity of the appID. (Vinod Kumar Vavilapalli via sseth) YARN-30. Fixed tests verifying web-services to work on JDK7. (Thomas Graves via vinodkv) YARN-150. Fixes AppRejectedTransition does not unregister a rejected app-attempt from the ApplicationMasterService (Bikas Saha via sseth) YARN-140. Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler. (ahmed via tucu) YARN-179. Fix some unit test failures. (Vinod Kumar Vavilapalli via sseth) YARN-181. Fixed eclipse settings broken by capacity-scheduler.xml move via YARN-140. (Siddharth Seth via vinodkv) YARN-169. Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter (Anthony Rojas via tomwhite) YARN-184. Remove unnecessary locking in fair scheduler, and address findbugs excludes. (sandyr via tucu) YARN-224. Fair scheduler logs too many nodeUpdate INFO messages. (Sandy Ryza via tomwhite) YARN-222. Fair scheduler should create queue for each user by default. (Sandy Ryza via tomwhite) MAPREDUCE-4778. Fair scheduler event log is only written if directory exists on HDFS. (Sandy Ryza via tomwhite) YARN-229. Remove old unused RM recovery code. (Bikas Saha via acmurthy) YARN-187. Add hierarchical queues to the fair scheduler. (Sandy Ryza via tomwhite) YARN-72. NM should handle cleaning up containers when it shuts down. (Sandy Ryza via tomwhite) YARN-267. Fix fair scheduler web UI. (Sandy Ryza via tomwhite) YARN-264. y.s.rm.DelegationTokenRenewer attempts to renew token even after removing an app. (kkambatl via tucu) YARN-271. Fair scheduler hits IllegalStateException trying to reserve different apps on same node. (Sandy Ryza via tomwhite) YARN-272. Fair scheduler log messages try to print objects without overridden toString methods. (sandyr via tucu) YARN-278. Fair scheduler maxRunningApps config causes no apps to make progress. (sandyr via tucu) YARN-282. Fair scheduler web UI double counts Apps Submitted. (sandyr via tucu) YARN-283. Fair scheduler fails to get queue info without root prefix. (sandyr via tucu) YARN-192. Node update causes NPE in the fair scheduler. (Sandy Ryza via tomwhite) YARN-288. Fair scheduler queue doesn't accept any jobs when ACLs are configured. (Sandy Ryza via tomwhite) YARN-300. After YARN-271, fair scheduler can infinite loop and not schedule any application. (Sandy Ryza via tomwhite) YARN-301. Fair scheduler throws ConcurrentModificationException when iterating over app's priorities. (Sandy Ryza via tomwhite) YARN-217. Fix RMAdmin protocol description to make it work in secure mode also. (Devaraj K via vinodkv) YARN-253. Fixed container-launch to not fail when there are no local resources to localize. (Tom White via vinodkv) YARN-330. Fix flakey test: TestNodeManagerShutdown#testKillContainersOnShutdown. (Sandy Ryza via hitesh) YARN-335. Fair scheduler doesn't check whether rack needs containers before assigning to node. (Sandy Ryza via tomwhite) YARN-336. Fair scheduler FIFO scheduling within a queue only allows 1 app at a time. (Sandy Ryza via tomwhite) YARN-135. Client tokens should be per app-attempt, and should be unregistered on App-finish. (vinodkv via sseth) YARN-302. Fair scheduler assignmultiple should default to false. (sandyr via tucu) YARN-372. Move InlineDispatcher from hadoop-yarn-server-resourcemanager to hadoop-yarn-common (sseth via hitesh) YARN-370. Fix SchedulerUtils to correctly round up the resource for containers. (Zhijie Shen via acmurthy) YARN-355. Fixes a bug where RM app submission could jam under load. (Daryn Sharp via sseth) Release 2.0.2-alpha - 2012-09-07 INCOMPATIBLE CHANGES YARN-9. Rename YARN_HOME to HADOOP_YARN_HOME. (vinodkv via acmurthy) NEW FEATURES YARN-1. Promote YARN to be a sub-project of Apache Hadoop. (acmurthy) IMPROVEMENTS YARN-29. Add a yarn-client module. (Vinod Kumar Vavilapalli via sseth) YARN-10. Fix DistributedShell module to not have a dependency on hadoop-mapreduce-client-core. (Hitesh Shah via vinodkv) YARN-80. Add support for delaying rack-local containers in CapacityScheduler. (acmurthy) YARN-137. Change the default YARN scheduler to be the CapacityScheduler. (sseth via acmurthy) OPTIMAZATIONS BUG FIXES YARN-12. Fix findbugs warnings in FairScheduler. (Junping Du via acmurthy) YARN-22. Fix ContainerLogs to work if the log-dir is specified as a URI. (Mayank Bansal via sseth) YARN-37. Change TestRMAppTransitions to use the DrainDispatcher. (Mayank Bansal via sseth) YARN-79. Implement close on all clients to YARN so that RPC clients don't throw exceptions on shut-down. (Vinod Kumar Vavilapalli) YARN-42. Modify NM's non-aggregating logs' handler to stop properly so that NMs don't get NPEs on startup errors. (Devaraj K via vinodkv) YARN-15. Updated default classpath for YARN applications to reflect split of YARN into a sub-project. (Arun C Murthy via vinodkv) YARN-75. Modified ResourceManager's RMContainer to handle a valid RELEASE event at RUNNING state. (Siddharth Seth via vinodkv) MAPREDUCE-2374. "Text File Busy" errors launching MR tasks. (Andy Isaacson via atm) YARN-138. Ensure default values for minimum/maximum container sizes is sane. (harsh & sseth via acmurthy) Release 0.23.10 - UNRELEASED INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS BUG FIXES Release 0.23.9 - 2013-07-08 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS YARN-427. Coverage fix for org.apache.hadoop.yarn.server.api.* (Aleksey Gorshkov via jeagles) YARN-478. fix coverage org.apache.hadoop.yarn.webapp.log (Aleksey Gorshkov via jeagles) OPTIMIZATIONS BUG FIXES Release 0.23.8 - 2013-06-05 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMIZATIONS YARN-548. Add tests for YarnUncaughtExceptionHandler (Vadim Bondarev via jeagles) BUG FIXES YARN-363. Add webapps/proxy directory without which YARN proxy-server fails when started in stand-alone mode. (Kenji Kikushima via vinodkv) YARN-690. RM exits on token cancel/renew problems (daryn via bobby) Release 0.23.7 - 2013-04-18 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS YARN-133 Update web services docs for RM clusterMetrics (Ravi Prakash via kihwal) YARN-249. Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x) (Ravi Prakash via tgraves) YARN-236. RM should point tracking URL to RM web page when app fails to start (Jason Lowe via jeagles) YARN-269. Resource Manager not logging the health_check_script result when taking it out (Jason Lowe via kihwal) YARN-227. Application expiration difficult to debug for end-users (Jason Lowe via jeagles) YARN-443. allow OS scheduling priority of NM to be different than the containers it launches (tgraves) YARN-468. coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter (Aleksey Gorshkov via bobby) YARN-200. yarn log does not output all needed information, and is in a binary format (Ravi Prakash via jlowe) YARN-525. make CS node-locality-delay refreshable (Thomas Graves via jlowe) OPTIMIZATIONS YARN-357. App submission should not be synchronized (daryn) BUG FIXES YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) YARN-364. AggregatedLogDeletionService can take too long to delete logs (jlowe) YARN-362. Unexpected extra results when using webUI table search (Ravi Prakash via jlowe) YARN-400. RM can return null application resource usage report leading to NPE in client (Jason Lowe via tgraves) YARN-426. Failure to download a public resource prevents further downloads (Jason Lowe via bobby) YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via bobby) YARN-345. Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager (Robert Parker via jlowe) YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal via bobby) YARN-460. CS user left in list of active users for the queue even when application finished (tgraves) Release 0.23.6 - 2013-02-06 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS YARN-285. Added a temporary plugin interface for RM to be able to redirect to JobHistory server for apps that it no longer tracks. (Derek Dagit via vinodkv) OPTIMIZATIONS BUG FIXES YARN-188. Coverage fixing for CapacityScheduler (Aleksey Gorshkov via bobby) YARN-214. RMContainerImpl does not handle event EXPIRE at state RUNNING (jeagles via bobby) YARN-151. Browser thinks RM main page JS is taking too long (Ravi Prakash via bobby) YARN-204. test coverage for org.apache.hadoop.tools (Aleksey Gorshkov via bobby) YARN-251. Proxy URI generation fails for blank tracking URIs (Tom White via jlowe) YARN-258. RM web page UI shows Invalid Date for start and finish times (Ravi Prakash via jlowe) YARN-266. RM and JHS Web UIs are blank because AppsBlock is not escaping string properly (Ravi Prakash via jlowe) YARN-280. RM does not reject app submission with invalid tokens (Daryn Sharp via tgraves) YARN-225. Proxy Link in RM UI thows NPE in Secure mode (Devaraj K via bobby) YARN-293. Node Manager leaks LocalizerRunner object for every Container (Robert Joseph Evans via jlowe) YARN-50. Implement renewal / cancellation of Delegation Tokens (Siddharth Seth via tgraves) YARN-320. RM should always be able to renew its own tokens. (Daryn Sharp via sseth) YARN-325. RM CapacityScheduler can deadlock when getQueueInfo() is called and a container is completing (Arun C Murthy via tgraves) YARN-334. Maven RAT plugin is not checking all source files (tgraves) YARN-354. WebAppProxyServer exits immediately after startup (Liang Xie via jlowe) Release 0.23.5 - 2012-11-28 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS YARN-161. Fix multiple compiler warnings for unchecked operations in YARN common. (Chris Nauroth via vinodkv) YARN-43. Fix TestResourceTrackerService to not depend on test order and thus pass on JDK7. (Thomas Graves via vinodkv) YARN-32. Fix TestApplicationTokens to not depend on test order and thus pass on JDK7. (vinodkv) YARN-186. Coverage fixing LinuxContainerExecutor (Aleksey Gorshkov via bobby) YARN-216. Remove jquery theming support. (Robert Joseph Evans via jlowe) OPTIMIZATIONS BUG FIXES YARN-163. Retrieving container log via NM webapp can hang with multibyte characters in log (jlowe via bobby) YARN-174. Modify NodeManager to pass the user's configuration even when rebooting. (vinodkv) YARN-177. CapacityScheduler - adding a queue while the RM is running has wacky results (acmurthy vai tgraves) YARN-178. Fix custom ProcessTree instance creation (Radim Kolar via bobby) YARN-180. Capacity scheduler - containers that get reserved create container token to early (acmurthy and bobby) YARN-139. Interrupted Exception within AsyncDispatcher leads to user confusion. (Vinod Kumar Vavilapalli via jlowe) YARN-165. RM should point tracking URL to RM web page for app when AM fails (jlowe via bobby) YARN-159. RM web ui applications page should be sorted to display last app first (tgraves via bobby) YARN-166. capacity scheduler doesn't allow capacity < 1.0 (tgraves via bobby) YARN-189. Fixed a deadlock between RM's ApplicationMasterService and the dispatcher. (Thomas Graves via vinodkv) YARN-202. Log Aggregation generates a storm of fsync() for namenode (Kihwal Lee via bobby) YARN-201. Fix CapacityScheduler to be less conservative for starved off-switch requests. (jlowe via acmurthy) YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. (jlowe via jeagles) YARN-212. NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't (Nathan Roberts via jlowe) YARN-219. NM should aggregate logs when application finishes. (bobby) Release 0.23.4 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS Change package of YarnClient to org.apache.hadoop. (Bikas Saha via vinodkv) YARN-108. FSDownload can create cache directories with the wrong permissions (Jason Lowe via bobby) OPTIMIZATIONS BUG FIXES YARN-88. DefaultContainerExecutor can fail to set proper permissions. (Jason Lowe via sseth) YARN-106. Nodemanager needs to set permissions of local directories (jlowe via bobby) Release 0.23.3 INCOMPATIBLE CHANGES NEW FEATURES IMPROVEMENTS OPTIMAZATIONS BUG FIXES YARN-14. Symlinks to peer distributed cache files no longer work (Jason Lowe via bobby) YARN-25. remove old aggregated logs (Robert Evans via tgraves) YARN-27. Failed refreshQueues due to misconfiguration prevents further refreshing of queues (Arun Murthy via tgraves) YARN-58. NM leaks filesystems (Jason Lowe via jeagles) YARN-39. RM-NM secret-keys should be randomly generated and rolled every so often. (vinodkv and sseth via sseth) YARN-31. Fix TestDelegationTokenRenewer to not depend on test order so as to pass tests on jdk7. (Thomas Graves via vinodkv) YARN-63. RMNodeImpl is missing valid transitions from the UNHEALTHY state (Jason Lowe via bobby) YARN-60. Fixed a bug in ResourceManager which causes all NMs to get NPEs and thus causes all containers to be rejected. (vinodkv) YARN-66. aggregated logs permissions not set properly (tgraves via bobby) YARN-68. NodeManager will refuse to shutdown indefinitely due to container log aggregation (daryn via bobby) YARN-87. NM ResourceLocalizationService does not set permissions of local cache directories (Jason Lowe via tgraves)