big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update CHANGES, RELEASENOTES, jdiff for 3.0.0 release.

Browse Source
This commit is contained in:
Andrew Wang 2017-12-13 13:05:32 -08:00
parent cb87e4dc92
commit d447152d49
4 changed files with 824 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

360
hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.0/CHANGES.3.0.0.md Normal file
View File

@ -0,0 +1,360 @@
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
# Apache Hadoop Changelog
## Release 3.0.0 - 2017-12-08
### INCOMPATIBLE CHANGES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-6623](https://issues.apache.org/jira/browse/YARN-6623) | Add support to turn off launching privileged containers in the container-executor | Blocker | nodemanager | Varun Vasudev | Varun Vasudev |
| [HADOOP-14816](https://issues.apache.org/jira/browse/HADOOP-14816) | Update Dockerfile to use Xenial | Major | build, test | Allen Wittenauer | Allen Wittenauer |
| [HADOOP-14957](https://issues.apache.org/jira/browse/HADOOP-14957) | ReconfigurationTaskStatus is exposing guava Optional in its public api | Major | common | Haibo Chen | Xiao Chen |
| [MAPREDUCE-6983](https://issues.apache.org/jira/browse/MAPREDUCE-6983) | Moving logging APIs over to slf4j in hadoop-mapreduce-client-core | Major | . | Jinjiang Ling | Jinjiang Ling |
| [HDFS-12682](https://issues.apache.org/jira/browse/HDFS-12682) | ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED | Blocker | erasure-coding | Xiao Chen | Xiao Chen |
### NEW FEATURES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-1492](https://issues.apache.org/jira/browse/YARN-1492) | truly shared cache for jars (jobjar/libjar) | Major | . | Sangjin Lee | Chris Trezzo |
| [HDFS-10467](https://issues.apache.org/jira/browse/HDFS-10467) | Router-based HDFS federation | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-5734](https://issues.apache.org/jira/browse/YARN-5734) | OrgQueue for easy CapacityScheduler queue configuration management | Major | . | Min Shen | Min Shen |
| [MAPREDUCE-5951](https://issues.apache.org/jira/browse/MAPREDUCE-5951) | Add support for the YARN Shared Cache | Major | . | Chris Trezzo | Chris Trezzo |
| [YARN-6871](https://issues.apache.org/jira/browse/YARN-6871) | Add additional deSelects params in RMWebServices#getAppReport | Major | resourcemanager, router | Giovanni Matteo Fumarola | Tanuj Nayak |
| [HADOOP-14840](https://issues.apache.org/jira/browse/HADOOP-14840) | Tool to estimate resource requirements of an application pipeline based on prior executions | Major | tools | Subru Krishnan | Rui Li |
| [YARN-3813](https://issues.apache.org/jira/browse/YARN-3813) | Support Application timeout feature in YARN. | Major | scheduler | nijel | Rohith Sharma K S |
### IMPROVEMENTS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-7045](https://issues.apache.org/jira/browse/YARN-7045) | Remove FSLeafQueue#addAppSchedulable | Major | fairscheduler | Yufei Gu | Sen Zhao |
| [YARN-7240](https://issues.apache.org/jira/browse/YARN-7240) | Add more states and transitions to stabilize the NM Container state machine | Major | . | Arun Suresh | kartheek muthyala |
| [HADOOP-14909](https://issues.apache.org/jira/browse/HADOOP-14909) | Fix the word of "erasure encoding" in the top page | Trivial | documentation | Takanobu Asanuma | Takanobu Asanuma |
| [HADOOP-14095](https://issues.apache.org/jira/browse/HADOOP-14095) | Document caveats about the default JavaKeyStoreProvider in KMS | Major | documentation, kms | Xiao Chen | Xiao Chen |
| [HADOOP-14928](https://issues.apache.org/jira/browse/HADOOP-14928) | Update site release notes for 3.0.0 GA | Major | site | Andrew Wang | Andrew Wang |
| [HDFS-12420](https://issues.apache.org/jira/browse/HDFS-12420) | Add an option to disallow 'namenode format -force' | Major | . | Ajay Kumar | Ajay Kumar |
| [HADOOP-14521](https://issues.apache.org/jira/browse/HADOOP-14521) | KMS client needs retry logic | Major | . | Rushabh S Shah | Rushabh S Shah |
| [YARN-2162](https://issues.apache.org/jira/browse/YARN-2162) | add ability in Fair Scheduler to optionally configure maxResources in terms of percentage | Major | fairscheduler, scheduler | Ashwin Shankar | Yufei Gu |
| [YARN-7207](https://issues.apache.org/jira/browse/YARN-7207) | Cache the RM proxy server address | Major | RM | Yufei Gu | Yufei Gu |
| [HADOOP-14939](https://issues.apache.org/jira/browse/HADOOP-14939) | Update project release notes with HDFS-10467 for 3.0.0 | Major | . | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12573](https://issues.apache.org/jira/browse/HDFS-12573) | Divide the total block metrics into replica and ec | Major | erasure-coding, metrics, namenode | Takanobu Asanuma | Takanobu Asanuma |
| [HDFS-12553](https://issues.apache.org/jira/browse/HDFS-12553) | Add nameServiceId to QJournalProtocol | Major | qjm | Bharat Viswanadham | Bharat Viswanadham |
| [HDFS-12603](https://issues.apache.org/jira/browse/HDFS-12603) | Enable async edit logging by default | Major | namenode | Andrew Wang | Andrew Wang |
| [HDFS-12642](https://issues.apache.org/jira/browse/HDFS-12642) | Log block and datanode details in BlockRecoveryWorker | Major | datanode | Xiao Chen | Xiao Chen |
| [HADOOP-14938](https://issues.apache.org/jira/browse/HADOOP-14938) | Configuration.updatingResource map should be initialized lazily | Major | . | Misha Dmitriev | Misha Dmitriev |
| [HDFS-12613](https://issues.apache.org/jira/browse/HDFS-12613) | Native EC coder should implement release() as idempotent function. | Major | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [MAPREDUCE-6972](https://issues.apache.org/jira/browse/MAPREDUCE-6972) | Enable try-with-resources for RecordReader | Major | . | Zoltan Haindrich | Zoltan Haindrich |
| [HADOOP-14880](https://issues.apache.org/jira/browse/HADOOP-14880) | [KMS] Document&test missing KMS client side configs | Minor | . | Wei-Chiu Chuang | Gabor Bota |
| [HDFS-12619](https://issues.apache.org/jira/browse/HDFS-12619) | Do not catch and throw unchecked exceptions if IBRs fail to process | Minor | namenode | Wei-Chiu Chuang | Wei-Chiu Chuang |
| [HADOOP-14771](https://issues.apache.org/jira/browse/HADOOP-14771) | hadoop-client does not include hadoop-yarn-client | Critical | common | Haibo Chen | Ajay Kumar |
| [YARN-7359](https://issues.apache.org/jira/browse/YARN-7359) | TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic | Minor | . | Haibo Chen | Haibo Chen |
| [HDFS-12448](https://issues.apache.org/jira/browse/HDFS-12448) | Make sure user defined erasure coding policy ID will not overflow | Major | erasure-coding | SammiChen | Huafeng Wang |
| [HADOOP-14944](https://issues.apache.org/jira/browse/HADOOP-14944) | Add JvmMetrics to KMS | Major | kms | Xiao Chen | Xiao Chen |
| [YARN-7261](https://issues.apache.org/jira/browse/YARN-7261) | Add debug message for better download latency monitoring | Major | nodemanager | Yufei Gu | Yufei Gu |
| [YARN-7357](https://issues.apache.org/jira/browse/YARN-7357) | Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations | Trivial | resourcemanager | Daniel Templeton | Sen Zhao |
| [YARN-4163](https://issues.apache.org/jira/browse/YARN-4163) | Audit getQueueInfo and getApplications calls | Major | . | Chang Li | Chang Li |
| [HADOOP-9657](https://issues.apache.org/jira/browse/HADOOP-9657) | NetUtils.wrapException to have special handling for 0.0.0.0 addresses and :0 ports | Minor | net | Steve Loughran | Varun Saxena |
| [YARN-7389](https://issues.apache.org/jira/browse/YARN-7389) | Make TestResourceManager Scheduler agnostic | Major | test | Robert Kanter | Robert Kanter |
| [HDFS-12544](https://issues.apache.org/jira/browse/HDFS-12544) | SnapshotDiff - support diff generation on any snapshot root descendant directory | Major | hdfs | Manoj Govindassamy | Manoj Govindassamy |
| [YARN-7358](https://issues.apache.org/jira/browse/YARN-7358) | TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler | Minor | resourcemanager | Haibo Chen | Haibo Chen |
| [YARN-7320](https://issues.apache.org/jira/browse/YARN-7320) | Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp\_ | Major | . | Misha Dmitriev | Misha Dmitriev |
| [YARN-7262](https://issues.apache.org/jira/browse/YARN-7262) | Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow | Major | . | Robert Kanter | Robert Kanter |
| [YARN-7397](https://issues.apache.org/jira/browse/YARN-7397) | Reduce lock contention in FairScheduler#getAppWeight() | Major | fairscheduler | Daniel Templeton | Daniel Templeton |
| [HADOOP-14992](https://issues.apache.org/jira/browse/HADOOP-14992) | Upgrade Avro patch version | Major | build | Chris Douglas | Bharat Viswanadham |
| [YARN-5326](https://issues.apache.org/jira/browse/YARN-5326) | Support for recurring reservations in the YARN ReservationSystem | Major | resourcemanager | Subru Krishnan | Carlo Curino |
| [YARN-6413](https://issues.apache.org/jira/browse/YARN-6413) | FileSystem based Yarn Registry implementation | Major | amrmproxy, api, resourcemanager | Ellen Hui | Ellen Hui |
| [HDFS-12482](https://issues.apache.org/jira/browse/HDFS-12482) | Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery | Minor | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [HDFS-12744](https://issues.apache.org/jira/browse/HDFS-12744) | More logs when short-circuit read is failed and disabled | Major | datanode | Weiwei Yang | Weiwei Yang |
| [HDFS-12771](https://issues.apache.org/jira/browse/HDFS-12771) | Add genstamp and block size to metasave Corrupt blocks list | Minor | . | Kuhu Shukla | Kuhu Shukla |
| [HADOOP-14987](https://issues.apache.org/jira/browse/HADOOP-14987) | Improve KMSClientProvider log around delegation token checking | Major | . | Xiaoyu Yao | Xiaoyu Yao |
| [MAPREDUCE-6975](https://issues.apache.org/jira/browse/MAPREDUCE-6975) | Logging task counters | Major | task | Prabhu Joseph | Prabhu Joseph |
| [YARN-7401](https://issues.apache.org/jira/browse/YARN-7401) | Reduce lock contention in ClusterNodeTracker#getClusterCapacity() | Major | resourcemanager | Daniel Templeton | Daniel Templeton |
| [HDFS-7060](https://issues.apache.org/jira/browse/HDFS-7060) | Avoid taking locks when sending heartbeats from the DataNode | Major | . | Haohui Mai | Jiandan Yang |
| [YARN-7413](https://issues.apache.org/jira/browse/YARN-7413) | Support resource type in SLS | Major | scheduler-load-simulator | Yufei Gu | Yufei Gu |
| [YARN-7386](https://issues.apache.org/jira/browse/YARN-7386) | Duplicate Strings in various places in Yarn memory | Major | . | Misha Dmitriev | Misha Dmitriev |
| [HADOOP-14960](https://issues.apache.org/jira/browse/HADOOP-14960) | Add GC time percentage monitor/alerter | Major | . | Misha Dmitriev | Misha Dmitriev |
| [HADOOP-15037](https://issues.apache.org/jira/browse/HADOOP-15037) | Add site release notes for OrgQueue and resource types | Major | . | Andrew Wang | Andrew Wang |
| [HADOOP-14876](https://issues.apache.org/jira/browse/HADOOP-14876) | Create downstream developer docs from the compatibility guidelines | Critical | documentation | Daniel Templeton | Daniel Templeton |
| [HADOOP-14112](https://issues.apache.org/jira/browse/HADOOP-14112) | Über-jira adl:// Azure Data Lake Phase I: Stabilization | Major | fs/adl | Steve Loughran | John Zhuge |
| [HADOOP-15104](https://issues.apache.org/jira/browse/HADOOP-15104) | AliyunOSS: change the default value of max error retry | Major | fs/oss | wujinhu | wujinhu |
### BUG FIXES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-7172](https://issues.apache.org/jira/browse/YARN-7172) | ResourceCalculator.fitsIn() should not take a cluster resource parameter | Major | scheduler | Daniel Templeton | Sen Zhao |
| [HADOOP-14901](https://issues.apache.org/jira/browse/HADOOP-14901) | ReuseObjectMapper in Hadoop Common | Minor | . | Hanisha Koneru | Hanisha Koneru |
| [YARN-7248](https://issues.apache.org/jira/browse/YARN-7248) | NM returns new SCHEDULED container status to older clients | Blocker | nodemanager | Jason Lowe | Arun Suresh |
| [HADOOP-14902](https://issues.apache.org/jira/browse/HADOOP-14902) | LoadGenerator#genFile write close timing is incorrectly calculated | Major | fs | Jason Lowe | Hanisha Koneru |
| [YARN-7084](https://issues.apache.org/jira/browse/YARN-7084) | TestSchedulingMonitor#testRMStarts fails sporadically | Major | . | Jason Lowe | Jason Lowe |
| [HDFS-12453](https://issues.apache.org/jira/browse/HDFS-12453) | TestDataNodeHotSwapVolumes fails in trunk Jenkins runs | Critical | test | Arpit Agarwal | Lei (Eddy) Xu |
| [HADOOP-14915](https://issues.apache.org/jira/browse/HADOOP-14915) | method name is incorrect in ConfServlet | Minor | . | Bharat Viswanadham | Bharat Viswanadham |
| [HADOOP-14752](https://issues.apache.org/jira/browse/HADOOP-14752) | TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky | Major | test | Andras Bokor | Andras Bokor |
| [HDFS-12569](https://issues.apache.org/jira/browse/HDFS-12569) | Unset EC policy logs empty payload in edit log | Blocker | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [YARN-6943](https://issues.apache.org/jira/browse/YARN-6943) | Update Yarn to YARN in documentation | Minor | documentation | Miklos Szegedi | Chetna Chaudhari |
| [YARN-7211](https://issues.apache.org/jira/browse/YARN-7211) | AMSimulator in SLS does't work due to refactor of responseId | Blocker | scheduler-load-simulator | Yufei Gu | Botong Huang |
| [HADOOP-14459](https://issues.apache.org/jira/browse/HADOOP-14459) | SerializationFactory shouldn't throw a NullPointerException if the serializations list is not defined | Minor | . | Nandor Kollar | Nandor Kollar |
| [YARN-7044](https://issues.apache.org/jira/browse/YARN-7044) | TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails | Major | capacity scheduler, test | Wangda Tan | Akira Ajisaka |
| [YARN-7226](https://issues.apache.org/jira/browse/YARN-7226) | Whitelisted variables do not support delayed variable expansion | Major | nodemanager | Jason Lowe | Jason Lowe |
| [HADOOP-14616](https://issues.apache.org/jira/browse/HADOOP-14616) | Client fails to read a block with erasure code (XOR, native) when one of the data block is lost | Blocker | . | Ayappan | Huafeng Wang |
| [YARN-7279](https://issues.apache.org/jira/browse/YARN-7279) | Fix typo in helper message of ContainerLauncher | Trivial | . | Elek, Marton | Elek, Marton |
| [YARN-7258](https://issues.apache.org/jira/browse/YARN-7258) | Add Node and Rack Hints to Opportunistic Scheduler | Major | . | Arun Suresh | kartheek muthyala |
| [YARN-7009](https://issues.apache.org/jira/browse/YARN-7009) | TestNMClient.testNMClientNoCleanupOnStop is flaky by design | Major | . | Miklos Szegedi | Miklos Szegedi |
| [HDFS-12567](https://issues.apache.org/jira/browse/HDFS-12567) | BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes | Major | erasure-coding | Andrew Wang | Andrew Wang |
| [HDFS-12494](https://issues.apache.org/jira/browse/HDFS-12494) | libhdfs SIGSEGV in setTLSExceptionStrings | Major | libhdfs | John Zhuge | John Zhuge |
| [YARN-7245](https://issues.apache.org/jira/browse/YARN-7245) | Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user | Major | capacity scheduler, yarn | Eric Payne | Eric Payne |
| [HDFS-12606](https://issues.apache.org/jira/browse/HDFS-12606) | When using native decoder, DFSStripedStream#close crashes JVM after being called multiple times. | Critical | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [HDFS-12599](https://issues.apache.org/jira/browse/HDFS-12599) | Remove Mockito dependency from DataNodeTestUtils | Minor | test | Ted Yu | Ted Yu |
| [YARN-7309](https://issues.apache.org/jira/browse/YARN-7309) | TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler | Major | test | Robert Kanter | Robert Kanter |
| [HADOOP-14912](https://issues.apache.org/jira/browse/HADOOP-14912) | FairCallQueue may defer servicing calls | Major | ipc | Daryn Sharp | Daryn Sharp |
| [HDFS-12635](https://issues.apache.org/jira/browse/HDFS-12635) | Unnecessary exception declaration of the CellBuffers constructor | Minor | . | Huafeng Wang | Huafeng Wang |
| [HDFS-12622](https://issues.apache.org/jira/browse/HDFS-12622) | Fix enumerate in HDFSErasureCoding.md | Minor | documentation | Akira Ajisaka | Yiqun Lin |
| [YARN-7082](https://issues.apache.org/jira/browse/YARN-7082) | TestContainerManagerSecurity failing in trunk | Major | . | Varun Saxena | Akira Ajisaka |
| [HADOOP-13556](https://issues.apache.org/jira/browse/HADOOP-13556) | Change Configuration.getPropsWithPrefix to use getProps instead of iterator | Major | . | Larry McCay | Larry McCay |
| [HADOOP-13102](https://issues.apache.org/jira/browse/HADOOP-13102) | Update GroupsMapping documentation to reflect the new changes | Major | documentation | Anu Engineer | Esther Kundin |
| [YARN-7270](https://issues.apache.org/jira/browse/YARN-7270) | Fix unsafe casting from long to int for class Resource and its sub-classes | Major | resourcemanager | Yufei Gu | Yufei Gu |
| [YARN-7310](https://issues.apache.org/jira/browse/YARN-7310) | TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler | Major | test | Robert Kanter | Robert Kanter |
| [YARN-7269](https://issues.apache.org/jira/browse/YARN-7269) | Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications | Critical | . | Sumana Sathish | Tan, Wangda |
| [HDFS-12659](https://issues.apache.org/jira/browse/HDFS-12659) | Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval | Minor | . | Ajay Kumar | Ajay Kumar |
| [HDFS-12485](https://issues.apache.org/jira/browse/HDFS-12485) | expunge may fail to remove trash from encryption zone | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang |
| [HADOOP-14949](https://issues.apache.org/jira/browse/HADOOP-14949) | TestKMS#testACLs fails intermittently | Major | kms, test | Xiao Chen | Xiao Chen |
| [YARN-7124](https://issues.apache.org/jira/browse/YARN-7124) | LogAggregationTFileController deletes/renames while file is open | Critical | nodemanager | Daryn Sharp | Jason Lowe |
| [YARN-7333](https://issues.apache.org/jira/browse/YARN-7333) | container-executor fails to remove entries from a directory that is not writable or executable | Critical | . | Jason Lowe | Jason Lowe |
| [YARN-7308](https://issues.apache.org/jira/browse/YARN-7308) | TestApplicationACLs fails with FairScheduler | Major | test | Robert Kanter | Robert Kanter |
| [HADOOP-14948](https://issues.apache.org/jira/browse/HADOOP-14948) | Document missing config key hadoop.treat.subject.external | Minor | security | Wei-Chiu Chuang | Ajay Kumar |
| [HDFS-12614](https://issues.apache.org/jira/browse/HDFS-12614) | FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured | Major | . | Manoj Govindassamy | Manoj Govindassamy |
| [YARN-7341](https://issues.apache.org/jira/browse/YARN-7341) | TestRouterWebServiceUtil#testMergeMetrics is flakey | Major | federation | Robert Kanter | Robert Kanter |
| [HDFS-12612](https://issues.apache.org/jira/browse/HDFS-12612) | DFSStripedOutputStream#close will throw if called a second time with a failed streamer | Major | erasure-coding | Andrew Wang | Lei (Eddy) Xu |
| [HADOOP-14958](https://issues.apache.org/jira/browse/HADOOP-14958) | CLONE - Fix source-level compatibility after HADOOP-11252 | Blocker | . | Junping Du | Junping Du |
| [YARN-7294](https://issues.apache.org/jira/browse/YARN-7294) | TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler | Major | . | Miklos Szegedi | Miklos Szegedi |
| [YARN-7170](https://issues.apache.org/jira/browse/YARN-7170) | Improve bower dependencies for YARN UI v2 | Critical | webapp | Sunil G | Sunil G |
| [YARN-7355](https://issues.apache.org/jira/browse/YARN-7355) | TestDistributedShell should be scheduler agnostic | Major | . | Haibo Chen | Haibo Chen |
| [HDFS-12497](https://issues.apache.org/jira/browse/HDFS-12497) | Re-enable TestDFSStripedOutputStreamWithFailure tests | Major | erasure-coding | Andrew Wang | Huafeng Wang |
| [HADOOP-14942](https://issues.apache.org/jira/browse/HADOOP-14942) | DistCp#cleanup() should check whether jobFS is null | Minor | . | Ted Yu | Andras Bokor |
| [YARN-7318](https://issues.apache.org/jira/browse/YARN-7318) | Fix shell check warnings of SLS. | Major | . | Wangda Tan | Gergely Novák |
| [HDFS-12518](https://issues.apache.org/jira/browse/HDFS-12518) | Re-encryption should handle task cancellation and progress better | Major | encryption | Xiao Chen | Xiao Chen |
| [HADOOP-14966](https://issues.apache.org/jira/browse/HADOOP-14966) | Handle JDK-8071638 for hadoop-common | Blocker | . | Bibin A Chundatt | Bibin A Chundatt |
| [HDFS-12249](https://issues.apache.org/jira/browse/HDFS-12249) | dfsadmin -metaSave to output maintenance mode blocks | Minor | namenode | Wei-Chiu Chuang | Wellington Chevreuil |
| [HDFS-12695](https://issues.apache.org/jira/browse/HDFS-12695) | Add a link to HDFS router federation document in site.xml | Minor | documentation | Yiqun Lin | Yiqun Lin |
| [YARN-7382](https://issues.apache.org/jira/browse/YARN-7382) | NoSuchElementException in FairScheduler after failover causes RM crash | Blocker | fairscheduler | Robert Kanter | Robert Kanter |
| [YARN-7385](https://issues.apache.org/jira/browse/YARN-7385) | TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE | Major | test | Robert Kanter | Yufei Gu |
| [HADOOP-14030](https://issues.apache.org/jira/browse/HADOOP-14030) | PreCommit TestKDiag failure | Major | security | John Zhuge | Wei-Chiu Chuang |
| [HADOOP-14979](https://issues.apache.org/jira/browse/HADOOP-14979) | Upgrade maven-dependency-plugin to 3.0.2 | Major | build | liyunzhang | liyunzhang |
| [HADOOP-14977](https://issues.apache.org/jira/browse/HADOOP-14977) | Xenial dockerfile needs ant in main build for findbugs | Trivial | build, test | Allen Wittenauer | Akira Ajisaka |
| [YARN-7339](https://issues.apache.org/jira/browse/YARN-7339) | LocalityMulticastAMRMProxyPolicy should handle cancel request properly | Minor | . | Botong Huang | Botong Huang |
| [HDFS-9914](https://issues.apache.org/jira/browse/HDFS-9914) | Fix configurable WebhDFS connect/read timeout | Blocker | hdfs-client, webhdfs | Xiaoyu Yao | Xiaoyu Yao |
| [YARN-7375](https://issues.apache.org/jira/browse/YARN-7375) | Possible NPE in RMWebapp when HA is enabled and the active RM fails | Major | . | Chandni Singh | Chandni Singh |
| [HDFS-12582](https://issues.apache.org/jira/browse/HDFS-12582) | Replace HdfsFileStatus constructor with a builder pattern. | Major | . | Bharat Viswanadham | Bharat Viswanadham |
| [HADOOP-14986](https://issues.apache.org/jira/browse/HADOOP-14986) | Enforce JDK limitations | Major | build | Chris Douglas | Chris Douglas |
| [HADOOP-14991](https://issues.apache.org/jira/browse/HADOOP-14991) | Add missing figures to Resource Estimator tool | Major | . | Subru Krishnan | Rui Li |
| [YARN-7299](https://issues.apache.org/jira/browse/YARN-7299) | Fix TestDistributedScheduler | Major | . | Jason Lowe | Arun Suresh |
| [YARN-6747](https://issues.apache.org/jira/browse/YARN-6747) | TestFSAppStarvation.testPreemptionEnable fails intermittently | Major | . | Sunil G | Miklos Szegedi |
| [YARN-7336](https://issues.apache.org/jira/browse/YARN-7336) | Unsafe cast from long to int Resource.hashCode() method | Critical | resourcemanager | Daniel Templeton | Miklos Szegedi |
| [YARN-7244](https://issues.apache.org/jira/browse/YARN-7244) | ShuffleHandler is not aware of disks that are added | Major | . | Kuhu Shukla | Kuhu Shukla |
| [HADOOP-14990](https://issues.apache.org/jira/browse/HADOOP-14990) | Clean up jdiff xml files added for 2.8.2 release | Blocker | . | Subru Krishnan | Junping Du |
| [HADOOP-14919](https://issues.apache.org/jira/browse/HADOOP-14919) | BZip2 drops records when reading data in splits | Critical | . | Aki Tanaka | Jason Lowe |
| [HDFS-12699](https://issues.apache.org/jira/browse/HDFS-12699) | TestMountTable fails with Java 7 | Major | . | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12714](https://issues.apache.org/jira/browse/HDFS-12714) | Hadoop 3 missing fix for HDFS-5169 | Major | native | Joe McDonnell | Joe McDonnell |
| [HDFS-12219](https://issues.apache.org/jira/browse/HDFS-12219) | Javadoc for FSNamesystem#getMaxObjects is incorrect | Trivial | . | Erik Krogen | Erik Krogen |
| [YARN-7412](https://issues.apache.org/jira/browse/YARN-7412) | test\_docker\_util.test\_check\_mount\_permitted() is failing | Critical | nodemanager | Haibo Chen | Eric Badger |
| [MAPREDUCE-6999](https://issues.apache.org/jira/browse/MAPREDUCE-6999) | Fix typo "onf" in DynamicInputChunk.java | Trivial | . | fang zhenyi | fang zhenyi |
| [YARN-7364](https://issues.apache.org/jira/browse/YARN-7364) | Queue dash board in new YARN UI has incorrect values | Critical | webapp | Sunil G | Sunil G |
| [YARN-7370](https://issues.apache.org/jira/browse/YARN-7370) | Preemption properties should be refreshable | Major | capacity scheduler, scheduler preemption | Eric Payne | Gergely Novák |
| [YARN-7400](https://issues.apache.org/jira/browse/YARN-7400) | incorrect log preview displayed in jobhistory server ui | Major | yarn | Santhosh B Gowda | Xuan Gong |
| [HADOOP-15013](https://issues.apache.org/jira/browse/HADOOP-15013) | Fix ResourceEstimator findbugs issues | Blocker | . | Allen Wittenauer | Arun Suresh |
| [YARN-7432](https://issues.apache.org/jira/browse/YARN-7432) | Fix DominantResourceFairnessPolicy serializable findbugs issues | Blocker | . | Allen Wittenauer | Daniel Templeton |
| [YARN-7434](https://issues.apache.org/jira/browse/YARN-7434) | Router getApps REST invocation fails with multiple RMs | Critical | . | Subru Krishnan | Íñigo Goiri |
| [HDFS-12725](https://issues.apache.org/jira/browse/HDFS-12725) | BlockPlacementPolicyRackFaultTolerant fails with very uneven racks | Major | erasure-coding | Xiao Chen | Xiao Chen |
| [YARN-5085](https://issues.apache.org/jira/browse/YARN-5085) | Add support for change of container ExecutionType | Major | . | Arun Suresh | Arun Suresh |
| [YARN-7428](https://issues.apache.org/jira/browse/YARN-7428) | Add containerId to Localizer failed logs | Minor | nodemanager | Prabhu Joseph | Prabhu Joseph |
| [YARN-7410](https://issues.apache.org/jira/browse/YARN-7410) | Cleanup FixedValueResource to avoid dependency to ResourceUtils | Major | resourcemanager | Sunil G | Wangda Tan |
| [YARN-7360](https://issues.apache.org/jira/browse/YARN-7360) | TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic | Major | test | Haibo Chen | Haibo Chen |
| [HADOOP-15018](https://issues.apache.org/jira/browse/HADOOP-15018) | Update JAVA\_HOME in create-release for Xenial Dockerfile | Blocker | build | Andrew Wang | Andrew Wang |
| [HDFS-12788](https://issues.apache.org/jira/browse/HDFS-12788) | Reset the upload button when file upload fails | Critical | ui, webhdfs | Brahma Reddy Battula | Brahma Reddy Battula |
| [YARN-7453](https://issues.apache.org/jira/browse/YARN-7453) | Fix issue where RM fails to switch to active after first successful start | Blocker | resourcemanager | Rohith Sharma K S | Rohith Sharma K S |
| [YARN-7458](https://issues.apache.org/jira/browse/YARN-7458) | TestContainerManagerSecurity is still flakey | Major | test | Robert Kanter | Robert Kanter |
| [HADOOP-15025](https://issues.apache.org/jira/browse/HADOOP-15025) | Ensure singleton for ResourceEstimatorService | Major | . | Subru Krishnan | Rui Li |
| [HDFS-12732](https://issues.apache.org/jira/browse/HDFS-12732) | Correct spellings of ramdomly to randomly in log. | Trivial | . | hu xiaodong | hu xiaodong |
| [YARN-7454](https://issues.apache.org/jira/browse/YARN-7454) | RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup | Minor | resourcemanager | Jason Lowe | Jason Lowe |
| [YARN-7388](https://issues.apache.org/jira/browse/YARN-7388) | TestAMRestart should be scheduler agnostic | Major | . | Haibo Chen | Haibo Chen |
| [YARN-7465](https://issues.apache.org/jira/browse/YARN-7465) | start-yarn.sh fails to start ResourceManager unless running as root | Blocker | . | Sean Mackrory | |
| [HADOOP-8522](https://issues.apache.org/jira/browse/HADOOP-8522) | ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used | Major | io | Mike Percy | Mike Percy |
| [YARN-7475](https://issues.apache.org/jira/browse/YARN-7475) | Fix Container log link in new YARN UI | Major | . | Sunil G | Sunil G |
| [HADOOP-15036](https://issues.apache.org/jira/browse/HADOOP-15036) | Update LICENSE.txt for HADOOP-14840 | Major | . | Arun Suresh | Arun Suresh |
| [YARN-6078](https://issues.apache.org/jira/browse/YARN-6078) | Containers stuck in Localizing state | Major | . | Jagadish | Billie Rinaldi |
| [YARN-7469](https://issues.apache.org/jira/browse/YARN-7469) | Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit | Major | capacity scheduler, yarn | Eric Payne | Eric Payne |
| [HADOOP-15067](https://issues.apache.org/jira/browse/HADOOP-15067) | GC time percentage reported in JvmMetrics should be a gauge, not counter | Major | . | Misha Dmitriev | Misha Dmitriev |
| [YARN-7290](https://issues.apache.org/jira/browse/YARN-7290) | Method canContainerBePreempted can return true when it shouldn't | Major | fairscheduler | Steven Rand | Steven Rand |
| [HDFS-12754](https://issues.apache.org/jira/browse/HDFS-12754) | Lease renewal can hit a deadlock | Major | . | Kuhu Shukla | Kuhu Shukla |
| [YARN-6647](https://issues.apache.org/jira/browse/YARN-6647) | RM can crash during transitionToStandby due to InterruptedException | Critical | resourcemanager | Jason Lowe | Bibin A Chundatt |
| [HDFS-11754](https://issues.apache.org/jira/browse/HDFS-11754) | Make FsServerDefaults cache configurable. | Minor | . | Rushabh S Shah | Mikhail Erofeev |
| [YARN-7509](https://issues.apache.org/jira/browse/YARN-7509) | AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby | Critical | . | Tao Yang | Tao Yang |
| [YARN-7589](https://issues.apache.org/jira/browse/YARN-7589) | TestPBImplRecords fails with NullPointerException | Major | . | Jason Lowe | Daniel Templeton |
| [HADOOP-15058](https://issues.apache.org/jira/browse/HADOOP-15058) | create-release site build outputs dummy shaded jars due to skipShade | Blocker | . | Andrew Wang | Andrew Wang |
| [YARN-7381](https://issues.apache.org/jira/browse/YARN-7381) | Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml | Critical | . | Xuan Gong | Xuan Gong |
| [HDFS-12889](https://issues.apache.org/jira/browse/HDFS-12889) | Router UI is missing robots.txt file | Major | . | Bharat Viswanadham | Bharat Viswanadham |
| [HDFS-12872](https://issues.apache.org/jira/browse/HDFS-12872) | EC Checksum broken when BlockAccessToken is enabled | Critical | erasure-coding | Xiao Chen | Xiao Chen |
| [HDFS-11576](https://issues.apache.org/jira/browse/HDFS-11576) | Block recovery will fail indefinitely if recovery time \> heartbeat interval | Critical | datanode, hdfs, namenode | Lukas Majercak | Lukas Majercak |
| [YARN-7607](https://issues.apache.org/jira/browse/YARN-7607) | Remove the trailing duplicated timestamp in container diagnostics message | Minor | nodemanager | Weiwei Yang | Weiwei Yang |
| [HDFS-12840](https://issues.apache.org/jira/browse/HDFS-12840) | Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog | Blocker | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [HADOOP-15080](https://issues.apache.org/jira/browse/HADOOP-15080) | Aliyun OSS: update oss sdk from 2.8.1 to 2.8.3 to remove its dependency on Cat-x "json-lib" | Blocker | fs/oss | Chris Douglas | SammiChen |
| [HADOOP-15059](https://issues.apache.org/jira/browse/HADOOP-15059) | 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade | Blocker | security | Junping Du | Jason Lowe |
### SUB-TASKS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-4081](https://issues.apache.org/jira/browse/YARN-4081) | Add support for multiple resource types in the Resource class | Major | resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-4172](https://issues.apache.org/jira/browse/YARN-4172) | Extend DominantResourceCalculator to account for all resources | Major | resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-4715](https://issues.apache.org/jira/browse/YARN-4715) | Add support to read resource types from a config file | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-4829](https://issues.apache.org/jira/browse/YARN-4829) | Add support for binary units | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-4830](https://issues.apache.org/jira/browse/YARN-4830) | Add support for resource types in the nodemanager | Major | nodemanager | Varun Vasudev | Varun Vasudev |
| [YARN-5242](https://issues.apache.org/jira/browse/YARN-5242) | Update DominantResourceCalculator to consider all resource types in calculations | Major | resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-5586](https://issues.apache.org/jira/browse/YARN-5586) | Update the Resources class to consider all resource types | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-5216](https://issues.apache.org/jira/browse/YARN-5216) | Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM | Major | distributed-scheduling | Arun Suresh | Hitesh Sharma |
| [YARN-5951](https://issues.apache.org/jira/browse/YARN-5951) | Changes to allow CapacityScheduler to use configuration store | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-5946](https://issues.apache.org/jira/browse/YARN-5946) | Create YarnConfigurationStore interface and InMemoryConfigurationStore class | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-6232](https://issues.apache.org/jira/browse/YARN-6232) | Update resource usage and preempted resource calculations to take into account all resource types | Major | resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-5948](https://issues.apache.org/jira/browse/YARN-5948) | Implement MutableConfigurationManager for handling storage into configuration store | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-5952](https://issues.apache.org/jira/browse/YARN-5952) | Create REST API for changing YARN scheduler configurations | Major | . | Jonathan Hung | Jonathan Hung |
| [HDFS-10882](https://issues.apache.org/jira/browse/HDFS-10882) | Federation State Store Interface API | Major | fs | Jason Kace | Jason Kace |
| [YARN-6445](https://issues.apache.org/jira/browse/YARN-6445) | [YARN-3926] Performance improvements in resource profile branch with respect to SLS | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev |
| [HDFS-10630](https://issues.apache.org/jira/browse/HDFS-10630) | Federation State Store FS Implementation | Major | hdfs | Íñigo Goiri | Jason Kace |
| [YARN-5949](https://issues.apache.org/jira/browse/YARN-5949) | Add pluggable configuration ACL policy interface and implementation | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-6575](https://issues.apache.org/jira/browse/YARN-6575) | Support global configuration mutation in MutableConfProvider | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-5953](https://issues.apache.org/jira/browse/YARN-5953) | Create CLI for changing YARN configurations | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-6761](https://issues.apache.org/jira/browse/YARN-6761) | Fix build for YARN-3926 branch | Major | nodemanager, resourcemanager | Varun Vasudev | Varun Vasudev |
| [YARN-6786](https://issues.apache.org/jira/browse/YARN-6786) | ResourcePBImpl imports cleanup | Trivial | resourcemanager | Daniel Templeton | Yeliang Cang |
| [YARN-5292](https://issues.apache.org/jira/browse/YARN-5292) | NM Container lifecycle and state transitions to support for PAUSED container state. | Major | . | Hitesh Sharma | Hitesh Sharma |
| [HDFS-12223](https://issues.apache.org/jira/browse/HDFS-12223) | Rebasing HDFS-10467 | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [HDFS-10687](https://issues.apache.org/jira/browse/HDFS-10687) | Federation Membership State Store internal API | Major | hdfs | Íñigo Goiri | Jason Kace |
| [YARN-5947](https://issues.apache.org/jira/browse/YARN-5947) | Create LeveldbConfigurationStore class using Leveldb as backing store | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-6322](https://issues.apache.org/jira/browse/YARN-6322) | Disable queue refresh when configuration mutation is enabled | Major | . | Jonathan Hung | Jonathan Hung |
| [HDFS-11826](https://issues.apache.org/jira/browse/HDFS-11826) | Federation Namenode Heartbeat | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-6788](https://issues.apache.org/jira/browse/YARN-6788) | Improve performance of resource profile branch | Blocker | nodemanager, resourcemanager | Sunil G | Sunil G |
| [HDFS-10880](https://issues.apache.org/jira/browse/HDFS-10880) | Federation Mount Table State Store internal API | Major | fs | Jason Kace | Íñigo Goiri |
| [HDFS-10646](https://issues.apache.org/jira/browse/HDFS-10646) | Federation admin tool | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-6994](https://issues.apache.org/jira/browse/YARN-6994) | Remove last uses of Long from resource types code | Minor | resourcemanager | Daniel Templeton | Daniel Templeton |
| [YARN-6892](https://issues.apache.org/jira/browse/YARN-6892) | Improve API implementation in Resources and DominantResourceCalculator class | Major | nodemanager, resourcemanager | Sunil G | Sunil G |
| [HDFS-11554](https://issues.apache.org/jira/browse/HDFS-11554) | [Documentation] Router-based federation documentation | Minor | fs | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12312](https://issues.apache.org/jira/browse/HDFS-12312) | Rebasing HDFS-10467 (2) | Major | . | Íñigo Goiri | Íñigo Goiri |
| [YARN-6610](https://issues.apache.org/jira/browse/YARN-6610) | DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources | Critical | resourcemanager | Daniel Templeton | Daniel Templeton |
| [YARN-7030](https://issues.apache.org/jira/browse/YARN-7030) | Performance optimizations in Resource and ResourceUtils class | Critical | nodemanager, resourcemanager | Wangda Tan | Wangda Tan |
| [YARN-7042](https://issues.apache.org/jira/browse/YARN-7042) | Clean up unit tests after YARN-6610 | Major | test | Daniel Templeton | Daniel Templeton |
| [YARN-6789](https://issues.apache.org/jira/browse/YARN-6789) | Add Client API to get all supported resource types from RM | Major | nodemanager, resourcemanager | Sunil G | Sunil G |
| [YARN-6781](https://issues.apache.org/jira/browse/YARN-6781) | ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter | Minor | resourcemanager | Daniel Templeton | Yu-Tang Lin |
| [HDFS-10631](https://issues.apache.org/jira/browse/HDFS-10631) | Federation State Store ZooKeeper implementation | Major | fs | Íñigo Goiri | Jason Kace |
| [YARN-7067](https://issues.apache.org/jira/browse/YARN-7067) | Optimize ResourceType information display in UI | Critical | nodemanager, resourcemanager | Wangda Tan | Wangda Tan |
| [YARN-7039](https://issues.apache.org/jira/browse/YARN-7039) | Fix javac and javadoc errors in YARN-3926 branch | Major | nodemanager, resourcemanager | Sunil G | Sunil G |
| [YARN-7024](https://issues.apache.org/jira/browse/YARN-7024) | Fix issues on recovery in LevelDB store | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-7093](https://issues.apache.org/jira/browse/YARN-7093) | Improve log message in ResourceUtils | Trivial | nodemanager, resourcemanager | Sunil G | Sunil G |
| [YARN-6933](https://issues.apache.org/jira/browse/YARN-6933) | ResourceUtils.DISALLOWED\_NAMES check is duplicated | Major | resourcemanager | Daniel Templeton | Manikandan R |
| [YARN-5328](https://issues.apache.org/jira/browse/YARN-5328) | Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem | Major | resourcemanager | Subru Krishnan | Subru Krishnan |
| [HDFS-12384](https://issues.apache.org/jira/browse/HDFS-12384) | Fixing compilation issue with BanDuplicateClasses | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12335](https://issues.apache.org/jira/browse/HDFS-12335) | Federation Metrics | Major | fs | Giovanni Matteo Fumarola | Íñigo Goiri |
| [YARN-5330](https://issues.apache.org/jira/browse/YARN-5330) | SharingPolicy enhancements required to support recurring reservations in ReservationSystem | Major | resourcemanager | Subru Krishnan | Carlo Curino |
| [YARN-7072](https://issues.apache.org/jira/browse/YARN-7072) | Add a new log aggregation file format controller | Major | . | Xuan Gong | Xuan Gong |
| [YARN-7136](https://issues.apache.org/jira/browse/YARN-7136) | Additional Performance Improvement for Resource Profile Feature | Critical | nodemanager, resourcemanager | Wangda Tan | Wangda Tan |
| [YARN-7137](https://issues.apache.org/jira/browse/YARN-7137) | Move newly added APIs to unstable in YARN-3926 branch | Blocker | nodemanager, resourcemanager | Wangda Tan | Wangda Tan |
| [YARN-6059](https://issues.apache.org/jira/browse/YARN-6059) | Update paused container state in the NM state store | Blocker | . | Hitesh Sharma | Hitesh Sharma |
| [HDFS-12430](https://issues.apache.org/jira/browse/HDFS-12430) | Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218 | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-7194](https://issues.apache.org/jira/browse/YARN-7194) | Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat | Major | . | Xuan Gong | Xuan Gong |
| [YARN-6612](https://issues.apache.org/jira/browse/YARN-6612) | Update fair scheduler policies to be aware of resource types | Major | fairscheduler | Daniel Templeton | Daniel Templeton |
| [HDFS-12450](https://issues.apache.org/jira/browse/HDFS-12450) | Fixing TestNamenodeHeartbeat and support non-HA | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-6840](https://issues.apache.org/jira/browse/YARN-6840) | Implement zookeeper based store for scheduler configuration updates | Major | . | Wangda Tan | Jonathan Hung |
| [YARN-7046](https://issues.apache.org/jira/browse/YARN-7046) | Add closing logic to configuration store | Major | . | Jonathan Hung | Jonathan Hung |
| [HDFS-12381](https://issues.apache.org/jira/browse/HDFS-12381) | [Documentation] Adding configuration keys for the Router | Minor | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-7238](https://issues.apache.org/jira/browse/YARN-7238) | Documentation for API based scheduler configuration management | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-7252](https://issues.apache.org/jira/browse/YARN-7252) | Removing queue then failing over results in exception | Critical | . | Jonathan Hung | Jonathan Hung |
| [YARN-7251](https://issues.apache.org/jira/browse/YARN-7251) | Misc changes to YARN-5734 | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-7250](https://issues.apache.org/jira/browse/YARN-7250) | Update Shared cache client api to use URLs | Minor | . | Chris Trezzo | Chris Trezzo |
| [YARN-6509](https://issues.apache.org/jira/browse/YARN-6509) | Add a size threshold beyond which yarn logs will require a force option | Major | . | Siddharth Seth | Xuan Gong |
| [YARN-7259](https://issues.apache.org/jira/browse/YARN-7259) | Add size-based rolling policy to LogAggregationIndexedFileController | Major | . | Xuan Gong | Xuan Gong |
| [MAPREDUCE-6971](https://issues.apache.org/jira/browse/MAPREDUCE-6971) | Moving logging APIs over to slf4j in hadoop-mapreduce-client-app | Major | . | Jinjiang Ling | Jinjiang Ling |
| [YARN-6550](https://issues.apache.org/jira/browse/YARN-6550) | Capture launch\_container.sh logs to a separate log file | Major | . | Wangda Tan | Suma Shivaprasad |
| [HDFS-12580](https://issues.apache.org/jira/browse/HDFS-12580) | Rebasing HDFS-10467 after HDFS-12447 | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [YARN-6916](https://issues.apache.org/jira/browse/YARN-6916) | Moving logging APIs over to slf4j in hadoop-yarn-server-common | Major | . | Akira Ajisaka | Akira Ajisaka |
| [YARN-7219](https://issues.apache.org/jira/browse/YARN-7219) | Make AllocateRequestProto compatible with branch-2/branch-2.8 | Critical | yarn | Ray Chiang | Ray Chiang |
| [YARN-6975](https://issues.apache.org/jira/browse/YARN-6975) | Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router | Major | . | Yeliang Cang | Yeliang Cang |
| [YARN-2037](https://issues.apache.org/jira/browse/YARN-2037) | Add work preserving restart support for Unmanaged AMs | Major | resourcemanager | Karthik Kambatla | Botong Huang |
| [YARN-5329](https://issues.apache.org/jira/browse/YARN-5329) | Placement Agent enhancements required to support recurring reservations in ReservationSystem | Blocker | resourcemanager | Subru Krishnan | Carlo Curino |
| [YARN-6182](https://issues.apache.org/jira/browse/YARN-6182) | Fix alignment issues and missing information in new YARN UI's Queue page | Major | yarn-ui-v2 | Akhil PB | Akhil PB |
| [YARN-2960](https://issues.apache.org/jira/browse/YARN-2960) | Add documentation for the YARN shared cache | Major | . | Chris Trezzo | Chris Trezzo |
| [HDFS-12273](https://issues.apache.org/jira/browse/HDFS-12273) | Federation UI | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12577](https://issues.apache.org/jira/browse/HDFS-12577) | Rename Router tooling | Major | fs | Íñigo Goiri | Íñigo Goiri |
| [HDFS-12541](https://issues.apache.org/jira/browse/HDFS-12541) | Extend TestSafeModeWithStripedFile with a random EC policy | Major | erasure-coding, test | Takanobu Asanuma | Takanobu Asanuma |
| [YARN-7304](https://issues.apache.org/jira/browse/YARN-7304) | Merge YARN-5734 branch to branch-3.0 | Major | . | Xuan Gong | Xuan Gong |
| [HDFS-12547](https://issues.apache.org/jira/browse/HDFS-12547) | Extend TestQuotaWithStripedBlocks with a random EC policy | Major | erasure-coding, test | Takanobu Asanuma | Takanobu Asanuma |
| [YARN-7205](https://issues.apache.org/jira/browse/YARN-7205) | Log improvements for the ResourceUtils | Major | nodemanager, resourcemanager | Jian He | Sunil G |
| [HDFS-12637](https://issues.apache.org/jira/browse/HDFS-12637) | Extend TestDistributedFileSystemWithECFile with a random EC policy | Major | erasure-coding, test | Takanobu Asanuma | Takanobu Asanuma |
| [YARN-7275](https://issues.apache.org/jira/browse/YARN-7275) | NM Statestore cleanup for Container updates | Blocker | . | Arun Suresh | kartheek muthyala |
| [YARN-7311](https://issues.apache.org/jira/browse/YARN-7311) | Fix TestRMWebServicesReservation parametrization for fair scheduler | Blocker | fairscheduler, reservation system | Yufei Gu | Yufei Gu |
| [YARN-6546](https://issues.apache.org/jira/browse/YARN-6546) | SLS is slow while loading 10k queues | Major | scheduler-load-simulator | Yufei Gu | Yufei Gu |
| [YARN-7345](https://issues.apache.org/jira/browse/YARN-7345) | GPU Isolation: Incorrect minor device numbers written to devices.deny file | Major | . | Jonathan Hung | Jonathan Hung |
| [YARN-7338](https://issues.apache.org/jira/browse/YARN-7338) | Support same origin policy for cross site scripting prevention. | Major | yarn-ui-v2 | Vrushali C | Sunil G |
| [HDFS-12620](https://issues.apache.org/jira/browse/HDFS-12620) | Backporting HDFS-10467 to branch-2 | Major | . | Íñigo Goiri | Íñigo Goiri |
| [YARN-4090](https://issues.apache.org/jira/browse/YARN-4090) | Make Collections.sort() more efficient by caching resource usage | Major | fairscheduler | Xianyin Xin | Yufei Gu |
| [YARN-7353](https://issues.apache.org/jira/browse/YARN-7353) | Docker permitted volumes don't properly check for directories | Major | yarn | Eric Badger | Eric Badger |
| [YARN-6984](https://issues.apache.org/jira/browse/YARN-6984) | DominantResourceCalculator.isAnyMajorResourceZero() should test all resources | Major | scheduler | Daniel Templeton | Sunil G |
| [YARN-3661](https://issues.apache.org/jira/browse/YARN-3661) | Basic Federation UI | Major | nodemanager, resourcemanager | Giovanni Matteo Fumarola | Íñigo Goiri |
| [YARN-4827](https://issues.apache.org/jira/browse/YARN-4827) | Document configuration of ReservationSystem for FairScheduler | Blocker | capacity scheduler | Subru Krishnan | Yufei Gu |
| [YARN-5516](https://issues.apache.org/jira/browse/YARN-5516) | Add REST API for supporting recurring reservations | Major | resourcemanager | Sangeetha Abdu Jyothi | Sean Po |
| [YARN-6505](https://issues.apache.org/jira/browse/YARN-6505) | Define the strings used in SLS JSON input file format | Major | scheduler-load-simulator | Yufei Gu | Gergely Novák |
| [YARN-7178](https://issues.apache.org/jira/browse/YARN-7178) | Add documentation for Container Update API | Blocker | . | Arun Suresh | Arun Suresh |
| [YARN-7374](https://issues.apache.org/jira/browse/YARN-7374) | Improve performance of DRF comparisons for resource types in fair scheduler | Critical | fairscheduler | Daniel Templeton | Daniel Templeton |
| [YARN-6927](https://issues.apache.org/jira/browse/YARN-6927) | Add support for individual resource types requests in MapReduce | Major | resourcemanager | Daniel Templeton | Gergo Repas |
| [YARN-7407](https://issues.apache.org/jira/browse/YARN-7407) | Moving logging APIs over to slf4j in hadoop-yarn-applications | Major | . | Yeliang Cang | Yeliang Cang |
| [YARN-7379](https://issues.apache.org/jira/browse/YARN-7379) | Moving logging APIs over to slf4j in hadoop-yarn-client | Major | . | Yeliang Cang | Yeliang Cang |
| [YARN-7276](https://issues.apache.org/jira/browse/YARN-7276) | Federation Router Web Service fixes | Major | . | Íñigo Goiri | Íñigo Goiri |
| [HADOOP-14997](https://issues.apache.org/jira/browse/HADOOP-14997) | Add hadoop-aliyun as dependency of hadoop-cloud-storage | Minor | fs/oss | Genmao Yu | Genmao Yu |
| [YARN-7289](https://issues.apache.org/jira/browse/YARN-7289) | Application lifetime does not work with FairScheduler | Major | resourcemanager | Miklos Szegedi | Miklos Szegedi |
| [YARN-7286](https://issues.apache.org/jira/browse/YARN-7286) | Add support for docker to have no capabilities | Major | yarn | Eric Badger | Eric Badger |
| [HDFS-11467](https://issues.apache.org/jira/browse/HDFS-11467) | Support ErasureCoding section in OIV XML/ReverseXML | Blocker | tools | Wei-Chiu Chuang | Huafeng Wang |
| [YARN-7307](https://issues.apache.org/jira/browse/YARN-7307) | Allow client/AM update supported resource types via YARN APIs | Blocker | nodemanager, resourcemanager | Wangda Tan | Sunil G |
| [MAPREDUCE-6997](https://issues.apache.org/jira/browse/MAPREDUCE-6997) | Moving logging APIs over to slf4j in hadoop-mapreduce-client-hs | Major | . | Akira Ajisaka | Gergely Novák |
| [MAPREDUCE-7001](https://issues.apache.org/jira/browse/MAPREDUCE-7001) | Moving logging APIs over to slf4j in hadoop-mapreduce-client-shuffle | Trivial | . | Jinjiang Ling | Jinjiang Ling |
| [YARN-7166](https://issues.apache.org/jira/browse/YARN-7166) | Container REST endpoints should report resource types | Major | resourcemanager | Daniel Templeton | Daniel Templeton |
| [YARN-7143](https://issues.apache.org/jira/browse/YARN-7143) | FileNotFound handling in ResourceUtils is inconsistent | Major | resourcemanager | Daniel Templeton | Daniel Templeton |
| [YARN-7437](https://issues.apache.org/jira/browse/YARN-7437) | Rename PlacementSet and SchedulingPlacementSet | Major | . | Wangda Tan | Wangda Tan |
| [YARN-6909](https://issues.apache.org/jira/browse/YARN-6909) | Use LightWeightedResource when number of resource types more than two | Critical | resourcemanager | Daniel Templeton | Sunil G |
| [YARN-7406](https://issues.apache.org/jira/browse/YARN-7406) | Moving logging APIs over to slf4j in hadoop-yarn-api | Major | . | Yeliang Cang | Yeliang Cang |
| [YARN-7442](https://issues.apache.org/jira/browse/YARN-7442) | [YARN-7069] Limit format of resource type name | Blocker | nodemanager, resourcemanager | Wangda Tan | Wangda Tan |
| [YARN-7369](https://issues.apache.org/jira/browse/YARN-7369) | Improve the resource types docs | Major | docs | Daniel Templeton | Daniel Templeton |
| [HADOOP-14993](https://issues.apache.org/jira/browse/HADOOP-14993) | AliyunOSS: Override listFiles and listLocatedStatus | Major | fs/oss | Genmao Yu | Genmao Yu |
| [YARN-7430](https://issues.apache.org/jira/browse/YARN-7430) | Enable user re-mapping for Docker containers by default | Blocker | security, yarn | Eric Yang | Eric Yang |
| [HADOOP-15024](https://issues.apache.org/jira/browse/HADOOP-15024) | AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server | Major | fs, fs/oss | SammiChen | SammiChen |
| [YARN-7541](https://issues.apache.org/jira/browse/YARN-7541) | Node updates don't update the maximum cluster capability for resources other than CPU and memory | Critical | resourcemanager | Daniel Templeton | Daniel Templeton |
### OTHER:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|:---- |:---- | :--- |:---- |:---- |:---- |
| [YARN-7317](https://issues.apache.org/jira/browse/YARN-7317) | Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy | Minor | . | Botong Huang | Botong Huang |
| [HDFS-12847](https://issues.apache.org/jira/browse/HDFS-12847) | Regenerate editsStored and editsStored.xml in HDFS tests | Major | test | Lei (Eddy) Xu | Lei (Eddy) Xu |
| [HADOOP-13493](https://issues.apache.org/jira/browse/HADOOP-13493) | Compatibility Docs should clarify the policy for what takes precedence when a conflict is found | Critical | documentation | Robert Kanter | Daniel Templeton |

139
hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.0/RELEASENOTES.3.0.0.md Normal file
View File

@ -0,0 +1,139 @@
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
# Apache Hadoop 3.0.0 Release Notes
These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.
---
* [YARN-7219](https://issues.apache.org/jira/browse/YARN-7219) | *Critical* | **Make AllocateRequestProto compatible with branch-2/branch-2.8**
Change update\_requests field id to 7. This matches the current field id in branch-2 and branch-2.8.
---
* [YARN-1492](https://issues.apache.org/jira/browse/YARN-1492) | *Major* | **truly shared cache for jars (jobjar/libjar)**
The YARN Shared Cache provides the facility to upload and manage shared application resources to HDFS in a safe and scalable manner. YARN applications can leverage resources uploaded by other applications or previous runs of the same application without having to re-­upload and localize identical files multiple times. This will save network resources and reduce YARN application startup time.
---
* [HDFS-10467](https://issues.apache.org/jira/browse/HDFS-10467) | *Major* | **Router-based HDFS federation**
HDFS Router-based Federation adds a RPC routing layer that provides a federated view of multiple HDFS namespaces.
This is similar to the existing ViewFS and HDFS federation functionality, except the mount table is managed on the server-side by the routing layer rather than on the client.
This simplifies access to a federated cluster for existing HDFS clients.
See HDFS-10467 and the HDFS Router-based Federation documentation for more details.
---
* [YARN-5734](https://issues.apache.org/jira/browse/YARN-5734) | *Major* | **OrgQueue for easy CapacityScheduler queue configuration management**
<!-- markdown -->
The OrgQueue extension to the capacity scheduler provides a programmatic way to change configurations by providing a REST API that users can call to modify queue configurations. This enables automation of queue configuration management by administrators in the queue's `administer_queue` ACL.
---
* [MAPREDUCE-5951](https://issues.apache.org/jira/browse/MAPREDUCE-5951) | *Major* | **Add support for the YARN Shared Cache**
MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.
---
* [YARN-6623](https://issues.apache.org/jira/browse/YARN-6623) | *Blocker* | **Add support to turn off launching privileged containers in the container-executor**
A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information.
---
* [HADOOP-14816](https://issues.apache.org/jira/browse/HADOOP-14816) | *Major* | **Update Dockerfile to use Xenial**
This patch changes the default build and test environment in the following ways:
\* Switch from Ubuntu "Trusty" 14.04 to Ubuntu "Xenial" 16.04
\* Switch from Oracle JDK 8 to OpenJDK 8
\* Adds OpenJDK 9 to the build environment
---
* [HADOOP-14957](https://issues.apache.org/jira/browse/HADOOP-14957) | *Major* | **ReconfigurationTaskStatus is exposing guava Optional in its public api**
ReconfigurationTaskStatus' API scope is reduced to LimitedPrivate, and its dependency on com.google.com.base.Optional is replaced by java.util.Optional.
---
* [HADOOP-14840](https://issues.apache.org/jira/browse/HADOOP-14840) | *Major* | **Tool to estimate resource requirements of an application pipeline based on prior executions**
The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.
---
* [YARN-5326](https://issues.apache.org/jira/browse/YARN-5326) | *Major* | **Support for recurring reservations in the YARN ReservationSystem**
Add native support for recurring reservations (good till cancelled) to enable periodic allocations of the same resources.
---
* [MAPREDUCE-6983](https://issues.apache.org/jira/browse/MAPREDUCE-6983) | *Major* | **Moving logging APIs over to slf4j in hadoop-mapreduce-client-core**
In hadoop-mapreduce-client-core module, the type of some public LOG variables were changed from org.apache.commons.logging.Log to org.slf4j.Logger. In the public methods that accepts logger, the logger was changed from org.apache.commons.logging.Log to org.slf4j.Logger.
---
* [HDFS-12682](https://issues.apache.org/jira/browse/HDFS-12682) | *Blocker* | **ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED**
**WARNING: No release note provided for this change.**
---
* [YARN-5085](https://issues.apache.org/jira/browse/YARN-5085) | *Major* | **Add support for change of container ExecutionType**
This allows the Application Master to ask the Scheduler to change the ExecutionType of a running/allocated container.
---
* [HDFS-12840](https://issues.apache.org/jira/browse/HDFS-12840) | *Blocker* | **Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog**
Add ErasureCodingPolicyId to each OP\_ADD edit log op.
---
* [HADOOP-15059](https://issues.apache.org/jira/browse/HADOOP-15059) | *Blocker* | **3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade**
This change reverses the default delegation token format implemented by HADOOP-12563, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

324
hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff/Apache_Hadoop_HDFS_3.0.0.xml Normal file
View File

File diff suppressed because one or more lines are too long

2
hadoop-project-dist/pom.xml
View File

@ -145,7 +145,7 @@
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<jdiff.stable.api>2.8.2</jdiff.stable.api>
<jdiff.stable.api>3.0.0</jdiff.stable.api>
<jdiff.stability>-unstable</jdiff.stability>
<!-- Commented out for HADOOP-11776 -->
<!-- Uncomment param name="${jdiff.compatibility}" in javadoc doclet if compatibility is not empty -->