big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19,688 Commits 2 Branches 2 Tags 264 MiB
b67ae6d9d7
Commit Graph

5254 Commits

Author SHA1 Message Date
Brahma Reddy Battula
01bd6ab18f HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 2018-02-07 23:10:33 +05:30
Tsz-Wo Nicholas Sze
4d1bf33d0f HDFS-12947. Limit the number of Snapshots allowed to be created for a Snapshottable Directory. Contributed by Shashikant Banerjee 2018-02-06 14:46:44 -08:00
Anu Engineer
4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Wei-Chiu Chuang
2021f4bdce HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang. 2018-02-02 17:18:42 -08:00
Hanisha Koneru
0ef639235b HDFS-13073. Cleanup code in InterQJournalProtocol.proto. Contributed by Bharat Viswanadham. 2018-02-02 16:28:03 -08:00
Akira Ajisaka
4aef8bd2ef
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative 2018-02-02 14:34:07 +09:00
Yiqun Lin
b0627c891b HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 11:25:41 +08:00
Anu Engineer
aa45faf0b2 HDFS-12942. Synchronization issue in FSDataSetImpl#moveBlock. Contributed by Ajay Kumar. 2018-02-01 18:03:01 -08:00
Hanisha Koneru
dd50f53997 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham. 2018-02-01 12:28:17 -08:00
Xiaoyu Yao
b3ae11d597 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar. 2018-02-01 10:45:34 -08:00
Inigo Goiri
6ca7204ceb HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 2018-02-01 10:37:14 -08:00
Xiao Chen
ae2177d296 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge. 2018-01-31 22:50:25 -08:00
Xiaoyu Yao
af015c0b23 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar. 2018-01-31 22:39:51 -08:00
Hanisha Koneru
3ce2190b58 HDFS-13092. Reduce verbosity for ThrottledAsyncChecker#schedule. Contributed by Mukul Kumar Singh. 2018-01-31 11:05:17 -08:00
Weiwei Yang
2e7331ca26 HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 2018-01-31 07:25:03 +08:00
Yiqun Lin
dbb9dded33 HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 2018-01-30 12:12:08 +08:00
Kihwal Lee
fde95d463c HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah 2018-01-29 17:23:29 -06:00
Xiao Chen
b63dcd583f HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi. 2018-01-28 22:19:49 -08:00
Yiqun Lin
7721fff744 HDFS-13042. RBF: Heartbeat Router State. Contributed by Inigo Goiri. 2018-01-25 15:51:26 +08:00
Inigo Goiri
95743c672e HDFS-12772. RBF: Federation Router State State Store internal API. Contributed by Inigo Goiri. 2018-01-23 19:15:44 -08:00
Yiqun Lin
d95c13774e HDFS-12963. Error log level in ShortCircuitRegistry#removeShm. Contributed by hu xiaodong. 2018-01-24 10:43:36 +08:00
Hanisha Koneru
6347b2253d HDFS-13023. Journal Sync does not work on a secure cluster. Contributed by Bharat Viswanadham. 2018-01-22 16:02:32 -08:00
Chen Liang
62c9e7fa99 HDFS-13036. Reusing the volume storage ID obtained by replicaInfo. Contributed by liaoyuxiangqin. 2018-01-19 17:51:53 -08:00
Lei Xu
2ed9d61aad HDFS-13039. StripedBlockReader.createBlockReader leaks socket on IOException. (Lei (Eddy) Xu) 2018-01-19 16:40:58 -08:00
Yiqun Lin
9e4f52d323 HDFS-12973. RBF: Document global quota supporting in federation. Contributed by Yiqun Lin. 2018-01-19 14:18:19 +08:00
Vinayakumar B
09efdfe9e1 HDFS-9049. Make Datanode Netty reverse proxy port to be configurable. Contributed by Vinayakumar B. 2018-01-17 14:16:48 +05:30
Brahma Reddy Battula
880b9d24ff HDFS-8693. refreshNamenodes does not support adding a new standby to a running DN. Contributed by Ajith S. 2018-01-16 16:21:02 +05:30
Yiqun Lin
9afb8025d6 HDFS-12972. RBF: Display mount table quota info in Web UI and admin command. Contributed by Yiqun Lin. 2018-01-13 13:16:43 +08:00
Inigo Goiri
d5d6a0353b HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-01-12 16:25:39 -08:00
Arpit Agarwal
b278f7b293 HDFS-12984. BlockPoolSlice can leak in a mini dfs cluster. Contributed by Ajay Kumar. 2018-01-12 08:04:00 -08:00
Yiqun Lin
d98a2e6e23 HDFS-12934. RBF: Federation supports global quota. Contributed by Yiqun Lin. 2018-01-10 13:59:11 +08:00
Inigo Goiri
d9006d8a4e HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by Inigo Goiri. 2018-01-09 18:53:25 -08:00
Manoj Govindassamy
73ff09b79a HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 2018-01-08 15:34:00 -08:00
Wei Yan
2ee0d64ace HDFS-12945. Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods. Contributed by Wei Yan. 2018-01-08 14:25:39 -08:00
Yiqun Lin
bf5c948995 HDFS-11848. Enhance dfsadmin listOpenFiles command to list files under a given path. Contributed by Yiqun Lin. 2018-01-06 14:31:08 +08:00
Inigo Goiri
83b513ac6d HDFS-12988. RBF: Mount table entries not properly updated in the local cache. Contributed by Inigo Goiri. 2018-01-05 09:11:38 -08:00
Arpit Agarwal
739d3c394d HDFS-12987. Document - Disabling the Lazy persist file scrubber.. Contributed by Karthik Palanisamy. 2018-01-04 10:32:00 -08:00
Yiqun Lin
2a48b3594c HDFS-12948. DiskBalancer report command top option should only take positive numeric values. Contributed by Shashikant Banerjee. 2018-01-04 10:48:44 +08:00
Manoj Govindassamy
5c28804b95 HDFS-12629. NameNode UI should report total blocks count by type - replicated and erasure coded. 2018-01-02 15:05:27 -08:00
Manoj Govindassamy
42a1c98597 HDFS-11847. Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning. 2018-01-02 14:59:36 -08:00
Lei Xu
6e3e1b8cde HDFS-12915. Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy. (Contributed by Chris Douglas) 2017-12-29 12:21:57 -08:00
Xiao Chen
5bf7e594d7 HDFS-9023. When NN is not able to identify DN for replication, reason behind it can be logged. 2017-12-28 11:54:08 -08:00
Chen Liang
826507c41b HDFS-12951. Incorrect javadoc in SaslDataTransferServer.java#receive. Contributed by Mukul Kumar Singh. 2017-12-21 11:20:30 -08:00
Wei Yan
a78db99190 HDFS-12932. Fix confusing LOG message for block replication. Contributed by Chao Sun. 2017-12-20 08:55:46 -08:00
Yiqun Lin
e040c97b77 HDFS-12937. RBF: Add more unit tests for router admin commands. Contributed by Yiqun Lin. 2017-12-19 15:31:34 +08:00
Virajith Jalaparti
8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Chris Douglas
2298f2d76b HDFS-12874. Documentation for provided storage. Contributed by Virajith Jalaparti 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
71ec170107 HDFS-12887. [READ] Allow Datanodes with Provided volumes to start when blocks with the same id exist locally 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti
fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
9c35be86e1 HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
a027055dd2 HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
b634053c4d HDFS-12591. [READ] Implement LevelDBFileRegionFormat. Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
352f994b64 HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
cc933cba77 HDFS-12685. [READ] FsVolumeImpl exception when scanning Provided storage volume 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
4d59dabb7f HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti
3d3be87e30 HDFS-12778. [READ] Report multiple locations for PROVIDED blocks 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
3b1d30301b HDFS-12775. [READ] Fix reporting of Provided volumes 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
e1a28f95b8 HDFS-12777. [READ] Reduce memory and CPU footprint for PROVIDED volumes. 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
6cd80b2521 HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
90d1b47a2a HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
71d0a82571 HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED 2017-12-15 17:51:39 -08:00
Virajith Jalaparti
98f5ed5aa3 HDFS-11902. [READ] Merge BlockFormatProvider and FileRegionProvider. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
d6a9a89973 HDFS-12605. [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
2407c9b93a HDFS-12093. [READ] Share remoteFS between ProvidedReplica instances. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
663b3c08b1 HDFS-12091. [READ] Check that the replicas served from a ProvidedVolumeImpl belong to the correct external storage 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
546b95f484 HDFS-11673. [READ] Handle failures of Datanode with PROVIDED storage 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
55ade54b8e HDFS-11792. [READ] Test cases for ProvidedVolumeDF and ProviderBlockIteratorImpl 2017-12-15 17:51:38 -08:00
Virajith Jalaparti
89b9faf529 HDFS-11703. [READ] Tests for ProvidedStorageMap 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
aa5ec85f7f HDFS-11663. [READ] Fix NullPointerException in ProvidedBlocksBuilder 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
1108cb7691 HDFS-11653. [READ] ProvidedReplica should return an InputStream that is bounded by its length 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
d65df0f273 HDFS-11190. [READ] Namenode support for data stored in external stores. 2017-12-15 17:51:37 -08:00
Virajith Jalaparti
b668eb9155 HDFS-10675. Datanode support to read from external stores. 2017-12-15 17:51:37 -08:00
Lei Xu
1c15b1751c HDFS-12819. Setting/Unsetting EC policy shows warning if the directory is not empty. (Contributed by Lei (Eddy) Xu) 2017-12-15 10:04:43 -08:00
Yiqun Lin
ee028bfdf1 HDFS-12895. RBF: Add ACL support for mount table. Contributed by Yiqun Lin. 2017-12-15 14:09:24 +08:00
Xiao Chen
e1cb278cd0 HDFS-12910. Secure Datanode Starter should log the port when it fails to bind. Contributed by Stephen O'Donnell and Nanda kumar. 2017-12-14 19:22:51 -08:00
Kihwal Lee
f5a72424c0 HDFS-12907. Allow read-only access to reserved raw for non-superusers. Contributed by Rushabh S Shah. 2017-12-14 14:58:47 -06:00
Jason Lowe
80db744ee5 HDFS-12881. Output streams closed with IOUtils suppressing write errors. Contributed by Ajay Kumar 2017-12-14 09:45:44 -06:00
Yiqun Lin
91c96bdf8e HDFS-12883. RBF: Document Router and State Store metrics. Contributed by Yiqun Lin. 2017-12-14 16:00:53 +08:00
Chen Liang
46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Chris Douglas
693169ef34 HDFS-12882. Support full open(PathHandle) contract in HDFS 2017-12-11 20:14:15 -08:00
Wei-Chiu Chuang
55fc2d6485 HDFS-12891. Do not invalidate blocks if toInvalidate is empty. Contributed by Zsolt Venczel. 2017-12-11 16:43:03 -08:00
Inigo Goiri
5cd1056ad7 HDFS-12875. RBF: Complete logic for -readonly option of dfsrouteradmin add command. Contributed by Inigo Goiri. 2017-12-11 15:14:57 -08:00
Manoj Govindassamy
ef7d334d36 HDFS-12825. Fsck report shows config key name for min replication issues (Contributed by Gabor Bota). 2017-12-08 11:46:30 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
f19638333b HADOOP-15059. Undoing the switch of Credentials to PB format as default - done via HADOOP-12563 for supporting 2.x to 3.x upgrades. 2017-12-08 08:00:21 -08:00
Wei-Chiu Chuang
d6c31a3e6b HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. Contributed by Vinayakumar B. 2017-12-07 14:50:30 -08:00
Lei Xu
67662e2ac9 HDFS-12840. Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog. Contributed by Lei (Eddy) Xu. 2017-12-07 11:15:40 -08:00
Akira Ajisaka
6cca5b3bcb
HDFS-12889. Addendum patch to add missing file. 2017-12-07 13:23:37 +09:00
Xiao Chen
56b1ff80dd HDFS-12872. EC Checksum broken when BlockAccessToken is enabled. 2017-12-05 20:48:16 -08:00
Brahma Reddy Battula
6d16a99ae8 HDFS-11751. DFSZKFailoverController daemon exits with wrong status code. Contributed by Bharat Viswanadham 2017-12-05 19:53:46 +05:30
Arpit Agarwal
e00c7f78c1 HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts. 2017-12-04 21:02:04 -08:00
Xiao Chen
404eab4dc0 HDFS-12396. Webhdfs file system should get delegation token from kms provider. Contributed by Rushabh S Shah. 2017-12-04 10:40:11 -08:00
Chris Douglas
42307e3c3a HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak 2017-12-01 22:34:30 -08:00
Wei-Chiu Chuang
0faf506245 HDFS-12836. startTxId could be greater than endTxId when tailing in-progress edit log. Contributed by Chao Sun. 2017-12-01 12:01:21 -08:00
Chris Douglas
53bbef3802 Revert "HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak" 
This reverts commit 5304698dc8.
 2017-12-01 11:19:38 -08:00
Chris Douglas
5304698dc8 HDFS-11576. Block recovery will fail indefinitely if recovery time > heartbeat interval. Contributed by Lukas Majercak 2017-12-01 10:29:30 -08:00
Konstantin V Shvachko
60fd0d7fd7 HDFS-12638. Delete copy-on-truncate block along with the original block, when deleting a file being truncated. Contributed by Konstantin Shvachko. 2017-11-30 18:18:28 -08:00
Tsz-Wo Nicholas Sze
b1c7654ee4 HDFS-12594. snapshotDiff fails if the report exceeds the RPC response limit. Contributed by Shashikant Banerjee 2017-11-30 12:18:29 -08:00
Chris Douglas
0e560f3b8d HDFS-12681. Make HdfsLocatedFileStatus a subtype of LocatedFileStatus 2017-11-29 20:28:06 -08:00
Kihwal Lee
53509f295b HDFS-11754. Make FsServerDefaults cache configurable. Contributed by Mikhail Erofeev. 2017-11-29 15:11:14 -06:00
Wei Yan
301641811d HDFS-12835. Fix the javadoc errors in Router-based federation. 2017-11-29 09:43:03 -08:00
Konstantin V Shvachko
d331762f24 HDFS-12832. INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit. Contribuited by Konstantin Shvachko. 2017-11-28 17:14:23 -08:00
Surendra Singh Lilhore
30941d99c9 HDFS-12857. StoragePolicyAdmin should support schema based path. Contributed by Surendra Singh Lilhore. 2017-11-28 23:57:03 +05:30
Chen Liang
60fc2a1388 HDFS-12804. Use slf4j instead of log4j in FSEditLog. Contributed by Mukul Kumar Singh. 2017-11-20 12:49:53 -08:00
Manoj Govindassamy
9fb4effd2c HDFS-12730. Verify open files captured in the snapshots across config disable and enable. 2017-11-20 10:28:12 -08:00
Inigo Goiri
e182e77794 HDFS-12801. RBF: Set MountTableResolver as default file resolver. Contributed by Inigo Goiri. 2017-11-16 16:58:47 -08:00
Weiwei Yang
462e25a3b2 HDFS-12814. Add blockId when warning slow mirror/disk in BlockReceiver. Contributed by Jiandan Yang. 2017-11-16 16:19:53 +08:00
Chris Douglas
675e9a8f57 Revert "HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus." 
This reverts commit b85603e3f8.
 2017-11-15 19:20:37 -08:00
Arpit Agarwal
6d201f77c7 HDFS-12498. Journal Syncer is not started in Federated + HA cluster. Contributed by Bharat Viswanadham. 2017-11-10 16:30:38 -08:00
Arpit Agarwal
1d6f8bebe9 HADOOP-14929. Cleanup usage of decodecomponent and use QueryStringDecoder from netty. Contributed by Bharat Viswanadham. 2017-11-10 16:28:12 -08:00
Chen Liang
10a1f557e7 HDFS-12791. NameNode Fsck http Connection can timeout for directories with multiple levels. Contributed by Mukul Kumar Singh. 2017-11-09 18:47:34 -08:00
Akira Ajisaka
3a3566e1d1
HDFS-12732. Correct spellings of ramdomly to randomly in log. Contributed by hu xiaodong. 2017-11-09 15:14:46 +09:00
Brahma Reddy Battula
410d0319cf HDFS-12788. Reset the upload button when file upload fails. Contributed by Brahma Reddy Battula 2017-11-08 14:41:16 +05:30
Weiwei Yang
bb8a6eea52 HDFS-7060. Avoid taking locks when sending heartbeats from the DataNode. Contributed by Jiandan Yang. 2017-11-08 10:22:13 +08:00
Xiaoyu Yao
169cdaa38e HDFS-10528. Add logging to successful standby checkpointing. Contributed by Xiaoyu Yao. 2017-11-03 20:34:40 -07:00
Chris Douglas
b85603e3f8 HDFS-12681. Fold HdfsLocatedFileStatus into HdfsFileStatus. 2017-11-03 14:30:57 -07:00
Kihwal Lee
4d2dce40bb HDFS-12771. Add genstamp and block size to metasave Corrupt blocks list. Contributed by Kuhu Shukla. 2017-11-03 15:16:46 -05:00
Xiao Chen
299d38295d HDFS-11467. Support ErasureCoding section in OIV XML/ReverseXML. Contributed by Huafeng Wang. 2017-11-03 12:05:45 -07:00
Xiao Chen
b00f828d84 HDFS-12725. BlockPlacementPolicyRackFaultTolerant fails with very uneven racks. 2017-11-02 21:53:13 -07:00
Xiao Chen
e565b5277d HDFS-12682. ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED. 2017-11-02 21:27:35 -07:00
Weiwei Yang
56b88b0670 HDFS-12744. More logs when short-circuit read is failed and disabled. Contributed by Weiwei Yang. 2017-11-01 16:41:45 +08:00
Yiqun Lin
20304b91cc HDFS-12219. Javadoc for FSNamesystem#getMaxObjects is incorrect. Contributed by Erik Krogen. 2017-11-01 14:37:08 +08:00
Lei Xu
9367c25dbd HDFS-12482. Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery. (lei) 2017-10-31 21:58:14 -07:00
Andrew Wang
5f681fa821 Revert "HDFS-12499. dfs.namenode.shared.edits.dir property is currently namenode specific key. Contributed by Bharat Viswanadham." 
This reverts commit b922ba7393.
 2017-10-31 10:46:10 -07:00
Arpit Agarwal
b922ba7393 HDFS-12499. dfs.namenode.shared.edits.dir property is currently namenode specific key. Contributed by Bharat Viswanadham. 2017-10-31 10:23:00 -07:00
Chris Douglas
d015e0bbd5 HDFS-7878. API - expose a unique file identifier. 2017-10-31 09:44:01 -07:00
Chris Douglas
d55a84951a HDFS-12582. Replace HdfsFileStatus constructor with a builder pattern. Contributed by Bharat Viswanadham 2017-10-27 15:36:03 -07:00
Arpit Agarwal
7da15eac7a HDFS-12579. JournalNodeSyncer should use fromUrl field of EditLogManifestResponse to construct servlet Url. Contributed by Hanisha Koneru. 2017-10-25 14:24:22 -07:00
Xiao Chen
f66ad1fab9 HADOOP-14957. ReconfigurationTaskStatus is exposing guava Optional in its public api. Contributed by Haibo Chen, Xiao Chen. 2017-10-25 14:11:30 -07:00
Manoj Govindassamy
075dd45a24 HDFS-12544. SnapshotDiff - support diff generation on any snapshot root descendant directory. 2017-10-25 10:54:40 -07:00
Zhe Zhang
17cd8d0c17 Revert "HDFS-12502. nntop should support a category based on FilesInGetListingOps." 
This reverts commit 60bfee270e.
 2017-10-24 15:06:50 -07:00
Wei-Chiu Chuang
0f2a69127a HDFS-12249. dfsadmin -metaSave to output maintenance mode blocks. Contributed by Wellington Chevreuil. 2017-10-23 15:24:34 -07:00
Arpit Agarwal
96be795656 HDFS-12650. Use slf4j instead of log4j in LeaseManager. Contributed by Ajay Kumar. 2017-10-23 13:20:46 -07:00
Arpit Agarwal
4b00c9a47f HDFS-12683. DFSZKFailOverController re-order logic for logging Exception. Contributed by Bharat Viswanadham. 2017-10-23 10:47:16 -07:00
Xiao Chen
248d9b6fff HDFS-12518. Re-encryption should handle task cancellation and progress better. 2017-10-20 16:25:04 -07:00
Kai Zheng
ce7cf66e5e HDFS-12448. Make sure user defined erasure coding policy ID will not overflow. Contributed by Huafeng Wang 2017-10-20 09:42:04 +08:00
Wei-Chiu Chuang
4ab0c8f96a HDFS-12619. Do not catch and throw unchecked exceptions if IBRs fail to process. Contributed by Wei-Chiu Chuang. 2017-10-19 06:17:59 -07:00
Zhe Zhang
60bfee270e HDFS-12502. nntop should support a category based on FilesInGetListingOps. 2017-10-18 23:51:24 -07:00
Lei Xu
31ebccc962 HDFS-12613. Native EC coder should implement release() as idempotent function. (Lei (Eddy) Xu) 2017-10-16 19:44:30 -07:00
Manoj Govindassamy
b406d8e375 HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. 2017-10-16 17:42:41 -07:00
Xiao Chen
21bc855587 HDFS-12642. Log block and datanode details in BlockRecoveryWorker. 2017-10-16 10:34:06 -07:00
Xiao Chen
035c6ee587 HDFS-12603. Enable async edit logging by default. Contributed by Andrew Wang. 2017-10-16 09:51:10 -07:00
Arpit Agarwal
8dd1eeb94f HDFS-12553. Add nameServiceId to QJournalProtocol. Contributed by Bharat Viswanadham 2017-10-13 14:22:21 -07:00
Arpit Agarwal
bb0a742aac HDFS-12627. Fix typo in DFSAdmin command output. Contributed by Ajay Kumar. 2017-10-11 12:26:14 -07:00
Manoj Govindassamy
78af6cdc53 HDFS-12573. Divide the total blocks metrics into replicated and erasure coded. Contributed by Takanobu Asanuma. 2017-10-10 14:23:29 -07:00
Jitendra Pandey
f29e55bf5e HADOOP-14910. Upgrade netty-all jar to latest 4.0.x.Final. Contributed by Vinayakumar B. 2017-10-10 11:00:07 -07:00
Andrew Wang
82cd85d868 Revert "HDFS-12603. Enable async edit logging by default." 
This reverts commit afb42aeabf.
 2017-10-09 17:53:56 -07:00
Andrew Wang
afb42aeabf HDFS-12603. Enable async edit logging by default. 2017-10-09 11:20:00 -07:00
Inigo Goiri
5d63a388d1 HDFS-12577. Rename Router tooling. Contributed by Inigo Goiri. 
(cherry picked from commit 53e8d0d030525e4c7f3875e23807c6dbe778890f)
 2017-10-06 18:50:52 -07:00
Inigo Goiri
81601dac8e HDFS-12273. Federation UI. Contributed by Inigo Goiri. 
(cherry picked from commit adbb2e00c7b85524fd43bd68895d49814c16680a)
 2017-10-06 18:50:52 -07:00
Inigo Goiri
6c69e23dcd HDFS-12580. Rebasing HDFS-10467 after HDFS-12447. Contributed by Inigo Goiri. 
(cherry picked from commit b12440d3479f19138bc66ea59baf41eb89061906)
 2017-10-06 18:50:51 -07:00
Inigo Goiri
ad41c81559 HDFS-12381. [Documentation] Adding configuration keys for the Router. Contributed by Inigo Goiri. 
(cherry picked from commit c2d6aa79055ef72406fa598e1c743b0c994b5da8)
 2017-10-06 18:50:51 -07:00
Inigo Goiri
67785fe006 HDFS-12450. Fixing TestNamenodeHeartbeat and support non-HA. Contributed by Inigo Goiri. 
(cherry picked from commit 928d1e87f9dbe64f89b858ccc1780723f3af58e7)
 2017-10-06 18:50:50 -07:00
Inigo Goiri
1f06b81ecb HDFS-12430. Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218. Contributed by Inigo Goiri. 
(cherry picked from commit 3302e792d469b7e8f3bfa10151e4e1c546589734)
 2017-10-06 18:50:50 -07:00
Inigo Goiri
bc9e588a19 HDFS-12335. Federation Metrics. Contributed by Inigo Goiri. 
(cherry picked from commit 3b19e77752afce87936f5c0d1e6d272fba798d7b)
 2017-10-06 18:50:50 -07:00
Inigo Goiri
cc58e7a983 HDFS-12384. Fixing compilation issue with BanDuplicateClasses. Contributed by Inigo Goiri. 
(cherry picked from commit fabe02c8fafa807198054da0c02b2ebaafda76aa)
 2017-10-06 18:50:49 -07:00
Inigo Goiri
7cb6bdf09e HDFS-10631. Federation State Store ZooKeeper implementation. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 23c4ddee11ab1300325a6361124ee8ad6f68d7a4)
 2017-10-06 18:50:49 -07:00
Inigo Goiri
346c9fce43 HDFS-12312. Rebasing HDFS-10467 (2). Contributed by Inigo Goiri. 
(cherry picked from commit 90ba6843fb3ac5dc7576535e66a75a5e3433247b)
 2017-10-06 18:50:48 -07:00
Inigo Goiri
b3e6bd22e3 HDFS-10646. Federation admin tool. Contributed by Inigo Goiri. 
(cherry picked from commit ae27e31fbcf546481db0b0345772db2e9132372e)
 2017-10-06 18:50:48 -07:00
Inigo Goiri
6f0de27318 HDFS-10880. Federation Mount Table State Store internal API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 58b97df661441150d35abd44b3a8606206b46441)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
d8c8107332 HDFS-11826. Federation Namenode Heartbeat. Contributed by Inigo Goiri. 
(cherry picked from commit 928f8dab52191e733984d37f47b69719ccf11313)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
55da7fd7eb HDFS-10687. Federation Membership State Store internal API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 95cae08849d23cf12c5d280c29f90908e6a90d40)
 2017-10-06 18:50:47 -07:00
Inigo Goiri
0ec82b8cdf HDFS-12223. Rebasing HDFS-10467. Contributed by Inigo Goiri. 
(cherry picked from commit 47db6e9d8e2c264671c89fdd6cb11a7c762d2cce)
 2017-10-06 18:50:46 -07:00
Inigo Goiri
ca4f209b49 HDFS-11546. Federation Router RPC server. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 8a9cdebebf26841a0f1e99fb08135f4597f2eba2)
 2017-10-06 18:50:46 -07:00
Inigo Goiri
4bf877b03f HDFS-10630. Federation State Store FS Implementation. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit c6e0bd640cdaf83a660fa050809cad6f1d4c6f4d)
 2017-10-06 18:50:45 -07:00
Inigo
2c740a684a HDFS-10882. Federation State Store Interface API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 6d94c90ece1c1d23d4c97e72c54e9991f5dbc481)
 2017-10-06 18:50:45 -07:00
Inigo
533b986633 HDFS-10881. Federation State Store Driver API. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 0f88e049156dce173afc0dbda864e29190dd2210)
 2017-10-06 18:50:45 -07:00
Inigo
2761bbc91a HDFS-10629. Federation Roter. Contributed by Jason Kace and Inigo Goiri. 
(cherry picked from commit 6821e801724ac38e9737538b2164c9ae88792282)
 2017-10-06 18:50:44 -07:00
Andrew Wang
644c2f6924 HDFS-12567. BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes. 2017-10-05 16:58:43 -07:00
Arpit Agarwal
b6942cbe9b HDFS-12420. Add an option to disallow 'namenode format -force'. Contributed by Ajay Kumar. 2017-10-05 15:26:52 -07:00
Xiaoyu Yao
107c177782 HDFS-12455. WebHDFS - Adding "snapshot enabled" status to ListStatus query result. Contributed by Ajay Kumar. 2017-10-03 13:02:12 -07:00
Arpit Agarwal
b91305119b HDFS-11968. ViewFS: StoragePolicies commands fail with HDFS federation. Contributed by Mukul Kumar Singh. 2017-10-03 11:23:40 -07:00
Anu Engineer
015abcd8ce HDFS-12576. JournalNodes are getting started, even though dfs.namenode.shared.edits.dir is not configured. Contributed by Bharat Viswanadham. 2017-10-02 15:40:00 -07:00
Lei Xu
27ffd43b64 HDFS-12569. Unset EC policy logs empty payload in edit log. (Lei (Eddy) Xu) 2017-10-02 15:35:49 -07:00
Chen Liang
5a68c2cef3 HDFS-12552. Use slf4j instead of log4j in FSNamesystem. Contributed by Ajay Kumar. 2017-10-02 11:26:53 -07:00
Daryn Sharp
0da29cbeea [HDFS-12386] Add fsserver defaults call to WebhdfsFileSystem. (Rushabh Shah via daryn) 2017-09-26 15:15:24 -05:00
Lei Xu
1267ff22ce HDFS-12523. Thread pools in ErasureCodingWorker do not shutdown. (Huafeng Wang via Lei) 2017-09-26 10:08:16 -07:00
Brahma Reddy Battula
9d3e4cccf9 HDFS-5040.Audit log for admin commands/ logging output of all DFS admin commands. Contributed by Kuhu Shukla. 2017-09-26 21:59:22 +05:30
Akira Ajisaka
02e2a9b115
HDFS-12304. Remove unused parameter from FsDatasetImpl#addVolume. Contributed by Chen Liang. 2017-09-26 01:25:07 +09:00
Anu Engineer
d0b2c5850b HDFS-12516. Suppress the fsnamesystem lock warning on nn startup. Contributed by Ajay Kumar. 2017-09-24 21:03:06 -07:00
Anu Engineer
415e5a157a HDFS-12375. Fail to start/stop journalnodes using start-dfs.sh/stop-dfs.sh. Contributed by Bharat Viswanadham. 2017-09-23 15:05:25 -07:00
Anu Engineer
40ef9fa171 HADOOP-14904. Fix javadocs issues in Hadoop HDFS. Contributed by Mukul Kumar Singh. 2017-09-23 12:16:48 -07:00
Anu Engineer
423ebcaf51 HDFS-12320. Add quantiles for transactions batched in Journal sync. Contributed by Hanisha Koneru. 2017-09-23 10:34:13 -07:00
Anu Engineer
36fb90c939 HDFS-12064. Reuse object mapper in HDFS. Contributed by Hanisha Koneru. 2017-09-23 10:11:00 -07:00
Jitendra Pandey
cda3378659 HDFS-12486. GetConf to get journalnodeslist. Contributed by Bharat Viswanadham. 2017-09-22 16:52:47 -07:00
Arpit Agarwal
08fca508e6 HDFS-12530. Processor argument in Offline Image Viewer should be case insensitive. Contributed by Hanisha Koneru. 2017-09-22 09:47:57 -07:00
Arpit Agarwal
b9e423fa8d HDFS-12496. Make QuorumJournalManager timeout properties configurable. Contributed by Ajay Kumar. 2017-09-21 08:44:43 -07:00
Kihwal Lee
6bf921a5c3 HDFS-12371. BlockVerificationFailures and BlocksVerified show up as 0 in Datanode JMX. Contributed by Hanisha Koneru. 2017-09-21 08:42:50 -05:00
Akira Ajisaka
50849ec9eb
HDFS-12526. FSDirectory should use Time.monotonicNow for durations. Contributed by Bharat Viswanadham. 2017-09-21 19:27:48 +09:00
Andrew Wang
a12f09ba3c HDFS-12447. Rename AddECPolicyResponse to AddErasureCodingPolicyResponse. Contributed by SammiChen. 2017-09-20 11:51:17 -07:00
Ming Ma
230b85d586 HDFS-12473. Change hosts JSON file format. 2017-09-20 09:03:59 -07:00
Andrew Wang
51edaacd09 HDFS-12445. Correct spellings of choosen to chosen. Contributed by hu xiaodong. 2017-09-19 13:48:23 -07:00
Brahma Reddy Battula
fda1221c55 HDFS-11799. Introduce a config to allow setting up write pipeline with fewer nodes than replication factor. Contributed by Brahma Reddy Battula 2017-09-19 11:25:45 +05:30
Kai Zheng
0f9af246e8 HDFS-12460. Make addErasureCodingPolicy an idempotent operation. Contributed by Sammi Chen 2017-09-18 18:07:12 +08:00
Erik Krogen
90894c7262 HDFS-12323. NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout. Contributed by Erik Krogen. 2017-09-15 13:56:27 -07:00
Lei Xu
fbe06b5880 HDFS-12349. Improve log message when it could not alloc enough blocks for EC. (Lei (Eddy) Xu) 2017-09-15 12:12:42 -07:00
Nathan Roberts
de197fc156 HDFS-12441. Supress UnresolvedPathException in namenode log. Contributed by Kihwal Lee. 2017-09-15 13:08:41 -05:00
Kihwal Lee
78bdf10ae4 HDFS-12424. Datatable sorting on the Datanode Information page in the Namenode UI is broken. Contributed by Shawna Martell. 2017-09-15 09:28:31 -05:00
Kai Zheng
08d996d3e9 HDFS-12395. Support erasure coding policy operations in namenode edit log. Contributed by Sammi Chen 2017-09-15 09:43:39 +08:00
Kai Zheng
ae8f55b932 HDFS-7859. Erasure Coding: Persist erasure coding policies in NameNode. Contributed by Sammi Chen 2017-09-15 09:08:18 +08:00
Lei Xu
61cee3a0b9 HDFS-12378. TestClientProtocolForPipelineRecovery#testZeroByteBlockRecovery fails on trunk. (Lei (Eddy) Xu) 2017-09-14 17:03:25 -07:00
Andrew Wang
390c2b5df0 Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang." 
This reverts commit 7fcc73fc0d.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2017-09-14 15:12:26 -07:00
Arpit Agarwal
65a941008d Revert "HDFS-10391. Always enable NameNode service RPC port. Contributed by Gergely Novak." 
This reverts commit aa4b6fbe75.
 2017-09-14 11:17:08 -07:00
Kai Zheng
e12f3e85bd HDFS-12438. Rename dfs.datanode.ec.reconstruction.stripedblock.threads.size to dfs.datanode.ec.reconstruction.threads. Contributed by Andrew Wang 2017-09-14 15:39:10 +08:00
ChenSammi
e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
Lei Xu
73aed34dff HDFS-12409. Add metrics of execution time of different stages in EC recovery task. (Lei (Eddy) Xu) 2017-09-13 17:14:13 -07:00
Lei Xu
123342cd07 HDFS-12412. Change ErasureCodingWorker.stripedReadPool to cached thread pool. (Lei (Eddy) Xu) 2017-09-12 18:12:07 -07:00
Arpit Agarwal
68282c8eac HDFS-12407. Journal node fails to shutdown cleanly if JournalNodeHttpServer or JournalNodeRpcServer fails to start. Contributed by Ajay Kumar. 2017-09-12 16:18:41 -07:00
Chen Liang
738c2a9682 HDFS-12406. dfsadmin command prints "Exception encountered" even if there is no exception, when debug is enabled. Contributed by Nandakumar. 2017-09-11 14:48:07 -07:00
Lei Xu
de9994bd89 Revert "HDFS-12349. Improve log message when it could not alloc enough blocks for EC. (lei)" 
This reverts commit 3e6d0ca2b2.
 2017-09-11 10:06:05 -07:00
Arpit Agarwal
aa4b6fbe75 HDFS-10391. Always enable NameNode service RPC port. Contributed by Gergely Novak. 2017-09-09 08:40:53 -07:00
Xiao Chen
b3a4d7d2a0 HDFS-12400. Provide a way for NN to drain the local key cache before re-encryption. 2017-09-07 20:51:02 -07:00
Lei Xu
3e6d0ca2b2 HDFS-12349. Improve log message when it could not alloc enough blocks for EC. (lei) 2017-09-07 18:01:37 -07:00
Manoj Govindassamy
3b3be355b3 HDFS-12404. Rename hdfs config authorization.provider.bypass.users to attributes.provider.bypass.users. 2017-09-07 17:20:42 -07:00
Andrew Wang
4e50dc976a HDFS-12218. Addendum. Rename split EC / replicated block metrics in BlockManager. 2017-09-07 16:57:19 -07:00
Andrew Wang
40c2f31f8d HDFS-12218. Rename split EC / replicated block metrics in BlockManager. 2017-09-07 16:56:35 -07:00
Xiao Chen
52b894db33 HDFS-12369. Edit log corruption due to hard lease recovery of not-closed file which has snapshots. 2017-09-07 16:30:12 -07:00
Yongjun Zhang
d77ed238a9 HDFS-12357. Let NameNode to bypass external attribute provider for configured users. Contributed by Yongjun Zhang, Arun Suresh. 2017-09-07 09:50:36 -07:00
Arpit Agarwal
5ff74167dd HDFS-12376. Enable JournalNode Sync by default. Contributed by Hanisha Koneru. 2017-09-07 08:18:28 -07:00
Kai Zheng
2adf8bed71 HDFS-12402. Refactor ErasureCodingPolicyManager and related codes. Contributed by Sammi Chen 2017-09-07 20:38:23 +08:00
Xiao Chen
0ba8ff4b77 HDFS-12359. Re-encryption should operate with minimum KMS ACL requirements. 2017-09-05 10:08:27 -07:00
Xiao Chen
633c1ea455 HDFS-12383. Re-encryption updater should handle canceled tasks better. 2017-09-03 19:50:41 -07:00
Xiao Chen
1b3b9938cf HDFS-12300. Audit-log delegation token related operations. 2017-08-31 23:20:01 -07:00
Mingliang Liu
1fbb662c70 HDFS-12363. Possible NPE in BlockManager$StorageInfoDefragmenter#scanAndCompactStorages. Contributed by Xiao Chen 2017-08-31 22:36:56 -07:00
Rakesh Radhakrishnan
200b11368d HDFS-12258. ec -listPolicies should list all policies in system, no matter it's enabled or disabled. Contributed by Wei Zhou. 2017-08-30 12:58:56 +05:30
Xiao Chen
f3661fd08e HDFS-12336. Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory. Contributed by Wellington Chevreuil. 2017-08-29 21:30:02 -07:00
Yongjun Zhang
cf93d60d3f HDFS-12191. Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done to this file. Contributed by Yongjun Zhang. 2017-08-29 16:57:03 -07:00
Arpit Agarwal
a1e3f84afe HDFS-12293. DataNode should log file name on disk error. Contributed by Ajay Kumar. 2017-08-28 15:49:59 -07:00
Arpit Agarwal
077a5eed9f HDFS-12358. Handle IOException when transferring edit log to Journal current dir through JN sync. Contributed by Hanisha Koneru. 2017-08-26 22:47:55 -07:00
Brahma Reddy Battula
bb6a3c8330 HDFS-12248. SNN will not upload fsimage on IOE and Interrupted exceptions. (Brahma Reddy Battula) 2017-08-26 13:47:12 +08:00
Manoj Govindassamy
b89ffcff36 HDFS-11986. Dfsadmin should report erasure coding related information separately. 2017-08-25 17:21:56 -07:00
Lei Xu
36bada3032 HDFS-12215. DataNode.transferBlock does not create its daemon in the xceiver thread group. (Lei Xu) 2017-08-25 16:01:14 -07:00
Andrew Wang
f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Arpit Agarwal
4b2c442d4e HDFS-12319. DirectoryScanner will throw IllegalStateException when Multiple BP's are present. Contributed by Brahma Reddy Battula. 2017-08-25 10:41:52 -07:00
Xiao Chen
1000a2af04 HDFS-10899. Add functionality to re-encrypt EDEKs. 2017-08-23 17:06:16 -07:00
Andrew Wang
b298948897 HDFS-12072. Provide fairness between EC and non-EC recovery tasks. Contributed by Eddy Xu. 2017-08-17 15:26:11 -07:00
Andrew Wang
96b3a6b972 HDFS-11082. Provide replicated EC policy to replicate files. Contributed by SammiChen. 2017-08-16 22:17:06 -07:00
Akira Ajisaka
08aaa4b36f
HDFS-12269. Better to return a Map rather than HashMap in getErasureCodingCodecs. Contributed by Huafeng Wang. 2017-08-17 13:20:27 +09:00
Ravi Prakash
f34646d652 HDFS-12301. NN File Browser UI: Navigate to a path when enter is pressed 2017-08-15 15:44:59 -07:00
Wei-Chiu Chuang
e3ae3e2644 HDFS-12066. When Namenode is in safemode,may not allowed to remove an user's erasure coding policy. Contributed by lufei. 2017-08-15 07:41:43 -07:00
Wei-Chiu Chuang
1040bae6fc HDFS-12054. FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode. Contributed by lufei. 2017-08-15 07:38:43 -07:00
Yiqun Lin
2e43c28e01 HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin. 2017-08-15 16:48:49 +08:00
Chris Douglas
645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
Lei Xu
ce797a1706 HDFS-12221. Replace xcerces in XmlEditsVisitor. (Ajay Kumar via lei) 2017-08-14 10:27:47 -07:00
Yiqun Lin
f13ca94954 HDFS-12287. Remove a no-longer applicable TODO comment in DatanodeManager. Contributed by Chen Liang. 2017-08-11 14:13:45 +08:00
John Zhuge
312e57b954 HDFS-11957. Enable POSIX ACL inheritance by default. Contributed by John Zhuge. 2017-08-10 10:30:47 -07:00
Kihwal Lee
b5c02f95b5 HDFS-12278. LeaseManager operations are inefficient in 2.8. Contributed by Rushabh S Shah. 2017-08-09 16:46:05 -05:00
Kihwal Lee
69afa26f19 HDFS-12157. Do fsyncDirectory(..) outside of FSDataset lock. Contributed by inayakumar B. 2017-08-09 09:03:51 -05:00
Wei-Chiu Chuang
9a3c2379ef HDFS-12182. BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks. Contributed by Wellington Chevreuil. 2017-08-08 23:44:18 -07:00
Kai Zheng
a53b8b6fdc HDFS-11975. Provide a system-default EC policy. Contributed by Huichun Lu 2017-08-09 10:12:58 +08:00
Haohui Mai
71b8dda4f6 HDFS-10326. Disable setting tcp socket send/receive buffers for write pipelines. Contributed by Daryn Sharp. 2017-08-08 14:58:16 -07:00
Kai Zheng
0b67436068 HDFS-12306. Add audit log for some erasure coding operations. Contributed by Huafeng Wang 2017-08-07 19:30:10 +08:00
Arpit Agarwal
bbc6d254c8 HDFS-12224. Add tests to TestJournalNodeSync for sync after JN downtime. Contributed by Hanisha Koneru. 2017-08-04 12:51:33 -07:00
Andrew Wang
f4c6b00a9f HDFS-12131. Add some of the FSNamesystem JMX values as metrics. Contributed by Erik Krogen. 2017-08-03 15:44:51 -07:00
Manoj Govindassamy
79df1e750e HDFS-9388. Decommission related code to support Maintenance State for datanodes. 2017-08-02 14:22:46 -07:00
Chris Douglas
12e44e7bda HDFS-6984. Serialize FileStatus via protobuf. 2017-08-02 12:12:48 -07:00
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Haohui Mai
6814324c33 HDFS-12107. FsDatasetImpl#removeVolumes floods the logs when removing the volume. Contributed by Kelvin Chu. 2017-08-01 18:34:07 -07:00
Manoj Govindassamy
52d7bafcf4 HDFS-12217. HDFS snapshots doesn't capture all open files when one of the open files is deleted. 2017-08-01 16:28:20 -07:00
Sean Mackrory
7774759830 HDFS-12151. Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes. 2017-08-01 14:34:25 -06:00
Weiwei Yang
ea568123fa HDFS-12154. Incorrect javadoc description in StorageLocationChecker#check. Contributed by Nandakumar. 2017-07-31 17:02:44 -07:00
Arpit Agarwal
3e23415a92 HDFS-12082. BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured. Contributed by Weiwei Yang. 2017-07-31 11:33:55 -07:00
Andrew Wang
480c8db40c HDFS-12206. Rename the split EC / replicated block metrics. 2017-07-28 11:24:44 -07:00
Lei Xu
77791e4c36 HDFS-12044. Mismatch between BlockManager.maxReplicationStreams and ErasureCodingWorker.stripedReconstructionPool pool size causes slow and bursty recovery. (Contributed by Lei (Eddy) Xu) 2017-07-28 10:50:49 -07:00
Wei-Chiu Chuang
369f731264 HDFS-12062. removeErasureCodingPolicy needs super user permission. Contributed by Wei-Chiu Chuang. 2017-07-28 00:51:03 -07:00
Brahma Reddy Battula
c4a85c694f HDFS-11896. Non-dfsUsed will be doubled on dead node re-registration. Contributed by Brahma Reddy Battula. 2017-07-27 12:02:57 -07:00
Kihwal Lee
a68b5b31cf HDFS-12171. Reduce IIP object allocations for inode lookup. Contributed by Daryn Sharp. 2017-07-25 11:03:49 -05:00
Kihwal Lee
1a79dcfc45 HDFS-12143. Improve performance of getting and removing inode features. Contributed by Daryn Sharp. 2017-07-25 10:28:57 -05:00
Akira Ajisaka
770cc46228
HDFS-12176. dfsadmin shows DFS Used%: NaN% if the cluster has zero block. Contributed by Weiwei Yang. 2017-07-24 16:24:23 +09:00
Kihwal Lee
8e3a992ecc HDFS-11742. Improve balancer usability after HDFS-8818. Contributed by Kihwal Lee 2017-07-21 09:14:19 -05:00
Konstantin V Shvachko
8c2c812832 HDFS-11472. Fix inconsistent replica size after a data pipeline failure. Contributed by Erik Krogen and Wei-Chiu Chuang. 2017-07-20 14:47:25 -07:00
Anu Engineer
413b23eb04 HDFS-12158. Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection. Contributed by Mukul Kumar Singh. 2017-07-19 10:29:06 -07:00
Brahma Reddy Battula
04ff412dab HDFS-12133. Correct ContentSummaryComputationContext Logger class name.. Contributed by Surendra Singh Lilhore. 2017-07-19 23:43:10 +08:00
Brahma Reddy Battula
f8cd55fe33 HDFS-12067. Correct dfsadmin commands usage message to reflects IPC port. Contributed by steven-wugang. 2017-07-19 23:21:43 +08:00
Tsz-Wo Nicholas Sze
ed27f2b2cc HDFS-12138. Remove redundant 'public' modifiers from BlockCollection. Contributed by Chen Liang 2017-07-17 13:54:16 -07:00
Tsz-Wo Nicholas Sze
f413ee33df HDFS-12130. Optimizing permission check for getContentSummary. Contributed by Chen Liang 2017-07-14 14:35:51 -07:00
Tsz-Wo Nicholas Sze
a1f12bb543 Revert "HDFS-12130. Optimizing permission check for getContentSummary." to fix commit message. 
This reverts commit a29fe100b3.
 2017-07-14 14:34:01 -07:00
Kihwal Lee
e7d187a1b6 HDFS-12140. Remove BPOfferService lock contention to get block pool id. Contributed by Daryn Sharp. 2017-07-14 16:07:17 -05:00
Kihwal Lee
8d86a93915 HDFS-12137. DN dataset lock should be fair. Contributed by Daryn Sharp. 2017-07-14 15:41:43 -05:00
Tsz-Wo Nicholas Sze
a29fe100b3 HDFS-12130. Optimizing permission check for getContentSummary. 2017-07-14 13:36:27 -07:00
Xiaoyu Yao
e15e2713e1 HDFS-11502. Datanode UI should display hostname based on JMX bean instead of window.location.hostname. Contributed by Jeffrey E Rodriguez and Yuanbo Liu. 2017-07-12 16:26:19 -07:00
Yiqun Lin
3a7f02b815 HDFS-12085. Reconfigure namenode heartbeat interval fails if the interval was set with time unit. Contributed by Weiwei Yang. 2017-07-11 20:40:11 +08:00
Akira Ajisaka
186650d21d
HDFS-12078. Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml. Contributed by Weiwei Yang. 2017-07-04 14:51:52 +09:00
Akira Ajisaka
b0560e0624
HDFS-12079. Description of dfs.block.invalidate.limit is incorrect in hdfs-default.xml. Contributed by Weiwei Yang. 2017-07-04 14:02:14 +09:00
Wei-Chiu Chuang
bcba844d11 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev. 2017-06-30 10:28:01 -07:00
Arpit Agarwal
6a9dc5f44b HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang. 2017-06-30 10:20:12 -07:00
Arpit Agarwal
a2f0cbd92f Revert "HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang." 
Accidentally committed the wrong patch version, reverting to fix that.

This reverts commit 900221f95e.
 2017-06-30 10:19:27 -07:00
Arpit Agarwal
900221f95e HDFS-12043. Add counters for block re-replication. Contributed by Chen Liang. 2017-06-29 17:15:13 -07:00
Andrew Wang
441378e7e4 Revert "HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin." 
This reverts commit 89a8edc014.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
 2017-06-29 13:54:16 -07:00
Wei-Chiu Chuang
16c8dbde57 HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy. 2017-06-29 06:38:41 -07:00
Wei-Chiu Chuang
f99b6d19de HDFS-11870. Add CLI cmd to enable/disable an erasure code policy. Contributed by lufei. 2017-06-28 10:54:25 -07:00
Anu Engineer
8641a2c08b HDFS-12045. Add log when Diskbalancer volume is transient storage type. Contributed by steven-wugang. 2017-06-27 00:39:47 -07:00
Lei Xu
144753e87f HDFS-12033. DatanodeManager picking EC recovery tasks should also consider the number of regular replication tasks. Contributed by Lei (Eddy) Xu. 2017-06-26 15:43:50 -07:00
Andrew Wang
2c367b464c HDFS-11956. Do not require a storage ID or target storage IDs when writing a block. Contributed by Ewan Higgs. 2017-06-26 11:20:07 -07:00
Andrew Wang
06c8ca3bb3 HDFS-12032. Inaccurate comment on DatanodeDescriptor#getNumberOfBlocksToBeErasureCoded. 2017-06-26 10:54:01 -07:00
Brahma Reddy Battula
abdea26280 HDFS-12024. Fix typo's in FsDatasetImpl.java. Contributed by Yasen liu. 2017-06-23 22:56:28 +08:00
Xiao Chen
68dc7c2405 HDFS-12009. Accept human-friendly units in dfsadmin -setBalancerBandwidth and -setQuota. Contributed by Andrew Wang. 2017-06-22 19:57:54 -07:00
Arpit Agarwal
9ae9467f92 HDFS-11998. Enable DFSNetworkTopology as default. Contributed by Chen Liang. 2017-06-21 22:01:41 -07:00
Zhe Zhang
c22cf00442 Confusion/name conflict between NameNodeActivity#BlockReportNumOps and RpcDetailedActivity#BlockReportNumOps. Contributed by Erik Krogen. 2017-06-21 16:34:19 -07:00
Kai Zheng
5db3f98468 HDFS-11606. Add CLI cmd to remove an erasure code policy. Contributed by Tim Yao 2017-06-21 14:01:55 +08:00
Akira Ajisaka
1a598479a9
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl. Contributed by hu xiaodong. 2017-06-21 11:19:48 +09:00
Lei Xu
45ff4d38e6 HDFS-11647. Add -E option in hdfs "count" command to show erasure policy summarization. Contributed by luhuichun. 2017-06-20 11:55:09 -07:00
Kihwal Lee
8c0769dee4 HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 2017-06-20 09:11:23 -05:00
Akira Ajisaka
2c769167db
HDFS-11345. Document the configuration key for FSNamesystem lock fairness. Contributed by Erik Krogen. 2017-06-20 16:35:54 +09:00
Brahma Reddy Battula
099cbb427a HDFS-11890. Handle NPE in BlockRecoveryWorker when DN is getting shoutdown. Contributed by Surendra Singh Lilhore. 2017-06-20 11:03:56 +08:00
Andrew Wang
fb68980959 HDFS-10480. Add an admin command to list currently open files. Contributed by Manoj Govindassamy. 2017-06-15 14:46:55 -07:00
Lei Xu
5e7cfdca7b HADOOP-14394. Provide Builder pattern for DistributedFileSystem.create. (lei) 2017-06-15 10:59:24 -07:00
Lei Xu
999c8fcbef HDFS-10999. Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats. (Manoj Govindassamy via lei) 2017-06-14 10:44:19 -07:00
Arpit Agarwal
8633ef8e10 HADOOP-14503. Make RollingAverages a mutable metric. Contributed by Hanisha Koneru. 2017-06-12 20:43:43 -07:00
Tsz-Wo Nicholas Sze
bec79ca249 HDFS-11947. When constructing a thread name, BPOfferService may print a bogus warning message. Contributed by Weiwei Yang 2017-06-13 09:45:10 +08:00
Arpit Agarwal
3f0a727f75 HDFS-11907. Add metric for time taken by NameNode resource check. Contributed by Chen Liang. 2017-06-12 15:18:38 -07:00
Mingliang Liu
1a33c9d589 HDFS-11945. Internal lease recovery may not be retried for a long time. Contributed by Kihwal Lee 2017-06-08 14:36:47 -07:00
Yiqun Lin
5ec71639cf HDFS-11929. Document missing processor of hdfs oiv_legacy command. Contributed by LiXin Ge. 2017-06-07 13:42:13 +08:00
Brahma Reddy Battula
1869e1771c HDFS-11711. DN should not delete the block On "Too many open files" Exception. Contributed by Brahma Reddy Battula. 2017-06-07 13:05:33 +08:00
Andrew Wang
c31cb879a3 HDFS-11840. Log HDFS Mover exception message of exit to its own log. Contributed by LiXin Ge. 2017-06-06 14:57:48 -07:00
Arpit Agarwal
867903db2b HDFS-11932. BPServiceActor thread name is not correctly set. Contributed by Chen Liang. 2017-06-06 13:51:02 -07:00
Yongjun Zhang
6a28a2b914 HDFS-11914. Add more diagnosis info for fsimage transfer failure. Contributed by Yongjun Zhang. 2017-06-05 16:31:03 -07:00
Arpit Agarwal
835560983e HDFS-11928. Segment overflow in FileDistributionCalculator. Contributed by LiXin Ge. 2017-06-05 13:21:22 -07:00
Yiqun Lin
8d9084eb62 HDFS-11359. DFSAdmin report command supports displaying maintenance state datanodes. Contributed by Yiqun Lin. 2017-06-02 12:48:30 +08:00
Xiao Chen
219f4c199e HDFS-11904. Reuse iip in unprotectedRemoveXAttrs calls. 2017-06-01 14:13:57 -07:00
Xiao Chen
6a3fc685a9 HDFS-11741. Long running balancer may fail due to expired DataEncryptionKey. Contributed by Wei-Chiu Chuang and Xiao Chen. 2017-06-01 14:05:37 -07:00
Kihwal Lee
1543d0f5be HDFS-5042. Completed files lost after power failure. Contributed by Vinayakumar B. 2017-05-31 10:55:03 -05:00
Brahma Reddy Battula
13de636b40 HDFS-11901. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin. 2017-05-31 23:09:08 +08:00
Akira Ajisaka
a7f085d6bf
HDFS-11832. Switch leftover logs to slf4j format in BlockManager.java. Contributed by Hui Xu and Chen Liang. 2017-05-29 17:30:23 +09:00
Inigo Goiri
303c8dc9b6 Fix NPE in LazyPersistFileScrubber. Contributed by Inigo Goiri. 2017-05-26 13:16:01 -07:00
Kihwal Lee
2b5ad48762 HDFS-11817. A faulty node can cause a lease leak and NPE on accessing data. Contributed by Kihwal Lee. 2017-05-25 17:17:38 -05:00
Arpit Agarwal
1161563136 HDFS-11879. Fix JN sync interval in case of exception. Contributed by Hanisha Koneru. 2017-05-25 14:01:53 -07:00
Kihwal Lee
29b7df960f HDFS-11856. Ability to re-add Upgrading Nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-25 13:05:23 -05:00
Arpit Agarwal
4fb41b31db HDFS-11878. Fix journal missing log httpServerUrl address in JournalNodeSyncer. Contributed by Hanisha Koneru. 2017-05-25 10:42:24 -07:00
Brahma Reddy Battula
2e41f8803d HDFS-11445. FSCK shows overall health stauts as corrupt even one replica is corrupt. Contributed by Brahma Reddy Battula. 2017-05-25 22:35:10 +08:00
Wei-Chiu Chuang
b8b69d797a Revert "HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory." 
This reverts commit 6a38d118d8.
 2017-05-24 17:21:22 -07:00
Wei-Chiu Chuang
2cba561228 Revert "HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang." 
This reverts commit bc7aff7cec.
 2017-05-24 17:20:27 -07:00
Arpit Agarwal
0e83ed5e73 HDFS-11877. FileJournalManager#getLogFile should ignore in progress edit logs during JN sync. Contributed by Hanisha Koneru. 2017-05-24 16:09:00 -07:00
Kai Zheng
a62be38a5e HDFS-11793. Allow to enable user defined erasure coding policy. Contributed by Sammi Chen 2017-05-24 18:45:52 +08:00
Rakesh Radhakrishnan
1b5451bf05 HDFS-11794. Add ec sub command -listCodec to show currently supported ec codecs. Contributed by SammiChen. 2017-05-23 17:03:28 +05:30
Arpit Agarwal
ca6bcc3c76 HDFS-11866. JournalNode Sync should be off by default in hdfs-default.xml. Contributed by Hanisha Koneru. 2017-05-22 17:53:47 -07:00
Brahma Reddy Battula
9cab42cc79 HDFS-11849. JournalNode startup failure exception should be logged in log file. Contributed By Surendra Singh Lilhore. 2017-05-23 02:39:19 +08:00
Arpit Agarwal
ec21ce425f HDFS-11827. NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method. Contributed by xupeng. 2017-05-16 19:51:04 -07:00
Akira Ajisaka
89a8edc014
HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin. 2017-05-16 12:41:59 -04:00
Kihwal Lee
9b90e52f1e HDFS-11641. Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus. Contributed by Daryn Sharp. 2017-05-16 11:28:46 -05:00
Vinayakumar B
1411612aa4 HDFS-11674. reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered (Contributed by Vinayakumar B) 2017-05-12 07:38:18 +05:30
Andrew Wang
83dd14aa84 HDFS-11757. Query StreamCapabilities when creating balancer's lock file. Contributed by SammiChen. 2017-05-11 14:09:49 -07:00
Chris Douglas
51b671ef18 HDFS-11681. DatanodeStorageInfo#getBlockIterator() should return an iterator to an unmodifiable set Contributed by Virajith Jalaparti 2017-05-10 22:25:28 -07:00
Kihwal Lee
ad1e3e4d9f HDFS-11755. Underconstruction blocks can be considered missing. Contributed by Nathan Roberts. 2017-05-10 14:15:57 -05:00
Andrew Wang
543aac9f28 HADOOP-14386. Rewind trunk from Guava 21.0 back to Guava 11.0.2. 2017-05-09 09:22:53 -07:00
Chris Douglas
a3954ccab1 HDFS-9807. Add an optional StorageID to writes. Contributed by Ewan Higgs 2017-05-05 12:01:26 -07:00
Yiqun Lin
97c2e576c9 HDFS-11530. Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault. Contributed by Yiqun Lin and Chen Liang. 2017-05-05 11:54:50 +08:00
Arpit Agarwal
07761af357 HDFS-11448. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-04 15:57:44 -07:00
Andrew Wang
c2a52ef9c2 HDFS-11643. Add shouldReplicate option to create builder. Contributed by SammiChen. 2017-05-04 11:39:14 -07:00
Arpit Agarwal
81092b1f11 HDFS-11722. Change Datanode file IO profiling sampling to percentage. Contributed by Hanisha Koneru. 2017-05-03 16:29:30 -07:00
Arpit Agarwal
30cd265134 Revert "HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru." 
This reverts commit 08fb82d6d1.
 2017-05-03 16:28:47 -07:00
Arpit Agarwal
08fb82d6d1 HDFS-11488. JN log segment syncing should support HA upgrade. Contributed by Hanisha Koneru. 2017-05-03 15:46:08 -07:00
Erik Krogen
d9014bda93 HDFS-11717. Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 2017-05-02 18:34:11 -07:00
Xiaoyu Yao
1058b40843 HDFS-11593. Change SimpleHttpProxyHandler#exceptionCaught log level from info to debug. Contributed by Xiaobing Zhou. 2017-05-01 16:12:51 -07:00
Kihwal Lee
4cfc866436 HDFS-11714. Newly added NN storage directory won't get initialized and cause space exhaustion. Contributed by Kihwal Lee. 2017-05-01 17:29:25 -05:00
Kihwal Lee
07b98e7830 HDFS-11609. Some blocks can be permanently lost if nodes are decommissioned while dead. Contributed by Kihwal Lee. 2017-05-01 14:19:02 -05:00
Kai Zheng
cb672a45a0 HDFS-11605. Allow user to customize new erasure code policies. Contributed by Huafeng Wang 2017-04-28 13:18:50 +08:00
Zhe Zhang
d8a3309830 HDFS-11709. StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 2017-04-27 09:50:30 -07:00
Konstantin V Shvachko
28eb2aabeb HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko. 2017-04-26 17:28:49 -07:00
Akira Ajisaka
93fa48fcf2
HDFS-11417. Add datanode admin command to get the storage info. Contributed by Surendra Singh Lilhore. 2017-04-26 17:43:29 +09:00
Chris Douglas
2f73396b59 HDFS-6708. StorageType should be encoded in the block token. Contributed by Ewan Higgs 2017-04-25 23:57:00 -07:00
Xiaoyu Yao
4ea2778acb HADOOP-14207. "dfsadmin -refreshCallQueue" fails with DecayRpcScheduler. Contributed by Surendra Singh Lihore. 2017-04-25 18:45:56 -07:00
Jason Lowe
e4321ec843 HDFS-11691. Add a proper scheme to the datanode links in NN web UI. Contributed by Kihwal Lee 2017-04-25 17:30:42 -05:00
Yongjun Zhang
20e3ae260b HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN files. (Manoj Govindassamy via Yongjun Zhang) 2017-04-21 20:35:48 -07:00
Kai Zheng
b0803388fc HDFS-11604. Define and parse erasure code policies. Contributed by Lin Zeng 2017-04-21 13:33:33 +08:00
Andrew Wang
74a723852d HDFS-11660. TestFsDatasetCache#testPageRounder fails intermittently with AssertionError. 2017-04-19 18:10:04 -07:00
Konstantin V Shvachko
8dfcd95d58 HDFS-11634. Optimize BlockIterator when interating starts in the middle. Contributed by Konstantin V Shvachko. 2017-04-17 16:56:24 -07:00
Zhe Zhang
ad49098eb3 HDFS-11615. FSNamesystemLock metrics can be inaccurate due to millisecond precision. Contributed by Erik Krogen. 2017-04-17 16:22:33 -07:00
Andrew Wang
a7312715a6 HDFS-10996. Ability to specify per-file EC policy at create time. Contributed by SammiChen. 2017-04-12 12:27:34 -07:00
Anu Engineer
abce613356 HDFS-11645. DataXceiver thread should log the actual error when getting InvalidMagicNumberException. Contributed by Chen Liang. 2017-04-12 11:40:58 -07:00
Kihwal Lee
d4c01dde49 HDFS-11648. Lazy construct the IIP pathname. Contributed by Daryn Sharp. 2017-04-12 13:29:24 -05:00
Chris Nauroth
23b1a7bdf1 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore. 2017-04-11 22:03:09 -07:00
Mingliang Liu
3a91376707 HDFS-11558. BPServiceActor thread name is too long. Contributed by Xiaobing Zhou 2017-04-11 15:29:38 -07:00
Chris Douglas
a5e57df3c5 HADOOP-14225. Remove xmlenc dependency 2017-04-10 11:48:40 -07:00
Andrew Wang
e8bdad7385 HDFS-11623. Move system erasure coding policies into hadoop-hdfs-client. 2017-04-07 16:46:28 -07:00
Andrew Wang
203edc026c HDFS-11633. FSImage failover disables all erasure coding policies. Contributed by Wei-Chiu Chuang. 2017-04-07 15:46:11 -07:00
Allen Wittenauer
96cbb4fce2 HADOOP-14202. fix jsvc/secure user var inconsistencies 
Signed-off-by: John Zhuge <jzhuge@apache.org>
 2017-04-07 08:59:21 -07:00
Xiaoyu Yao
a49fac5302 HDFS-11362. StorageDirectory should initialize a non-null default StorageDirType. Contribute by Hanisha Koneru. 2017-04-06 14:33:16 -07:00
Arpit Agarwal
8c57aeb5b4 HDFS-11131. TestThrottledAsyncChecker#testCancellation is flaky. 2017-04-05 17:54:30 -07:00
Andrew Wang
9e0e430f18 HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang. 2017-04-04 23:05:24 -07:00
Akira Ajisaka
8cb7325527
HDFS-9651. Add missing robots.txt files. 2017-04-05 12:47:45 +09:00
Andrew Wang
18432130a7 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-04 13:38:25 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Wei-Chiu Chuang
bc7aff7cec HDFS-11515. -du throws ConcurrentModificationException. Contributed by Istvan Fajth, Wei-Chiu Chuang. 2017-04-03 07:35:09 -07:00
Hanisha Koneru
73835c73e2 HDFS-11560. Expose slow disks via NameNode JMX. Contributed by Hanisha Koneru. 2017-03-31 13:50:29 -07:00
Arpit Agarwal
5485d93bda HDFS-11603. Improve slow mirror/disk warnings in BlockReceiver. 2017-03-31 12:10:27 -07:00
Hanisha Koneru
28cdc5a8dc HDFS-11551. Handle SlowDiskReport from DataNode at the NameNode. Contributed by Hanisha Koneru. 2017-03-30 22:41:26 -07:00
Mingliang Liu
e5f0622a6f HDFS-11592. Closing a file has a wasteful preconditions in NameNode. Contributed by Eric Badger 2017-03-30 15:44:21 -07:00
Daniel Templeton
15e3873dc3 HDFS-11571. Typo in DataStorage exception message (Contributed by Anna Budai via Daniel Templeton) 2017-03-29 10:56:36 -07:00
Rakesh Radhakrishnan
84d787b9d5 HDFS-11541. Call RawErasureEncoder and RawErasureDecoder release() methods. Contributed by SammiChen. 2017-03-29 11:41:48 +05:30
Yiqun Lin
6b09336438 HDFS-11577. Combine the old and the new chooseRandom for better performance. Contributed by Chen Liang. 2017-03-28 23:02:07 +08:00
Wei-Chiu Chuang
46d37a65cf HDFS-10506. Addendum patch to include missing changes. Contributed by Akira Ajisaka. 2017-03-27 05:23:45 -07:00
Yiqun Lin
96e2ab8144 HDFS-11555. Fix typos in class OfflineImageReconstructor. Contributed by Yiqun Lin. 2017-03-27 19:23:37 +08:00
Tsuyoshi Ozawa
84ddedc0b2 HADOOP-10101. Update guava dependency to the latest version. (ozawa) 2017-03-25 10:06:52 +09:00
Xiaobing Zhou
1168ece596 HDFS-11534. Add counters for number of blocks in pending IBR. Contributed by Xiaobing Zhou. 2017-03-24 14:33:44 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Ravi Prakash
d4f73e7e27 HDFS-11574. Spelling mistakes in the Java source. Contributed by Hu Xiaodong. 2017-03-24 09:38:17 -07:00
Yiqun Lin
3b908f71c5 HDFS-11358. DiskBalancer: Report command supports reading nodes from host file. Contributed by Yiqun Lin. 2017-03-21 17:44:17 +08:00
Arpit Agarwal
e7c8da614c HDFS-11545. Propagate DataNode's slow disks info to the NameNode via Heartbeats. Contributed by Hanisha Koneru. 2017-03-20 21:54:58 -07:00
Chris Douglas
b104f3a282 HDFS-6648. Order of namenodes in ConfiguredFailoverProxyProvider is undefined. Contributed by Inigo Goiri 2017-03-20 17:15:13 -07:00
Arpit Agarwal
ffa160ddb8 HDFS-11547. Add logs for slow BlockReceiver while writing data to disk. Contributed by Xiaobing Zhou. 2017-03-17 19:02:24 -07:00
Hanisha Koneru
7f8e928400 HDFS-11517. Expose slow disks via DataNode JMX. Contributed by Hanisha Koneru 2017-03-17 15:42:25 -07:00
Andrew Wang
92ea6d74ec HDFS-11516. Admin command line should print message to stderr in failure case. Contributed by Kai Sasaki. 2017-03-17 11:26:02 -07:00
Andrew Wang
4812518b23 HDFS-10530. BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy. Contributed by Manoj Govindassamy and Rui Gao. 2017-03-16 15:07:38 -07:00
Kihwal Lee
ba62b50eba HDFS-10601. Improve log message to include hostname when the NameNode is in safemode. Contributed by Kuhu Shukla. 2017-03-16 10:54:59 -05:00
Hanisha Koneru
d69a82c89c HDFS-11511. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-03-15 18:01:45 -07:00
Chen Liang
615ac09499 HDFS-11419. DFSTopologyNodeImpl#chooseRandom optimizations. Contributed by Chen Liang. 2017-03-15 12:28:22 -07:00
Yiqun Lin
4416a07c9c HDFS-11420. Edit file should not be processed by the same type processor in OfflineEditsViewer. Contributed by Yiqun Lin. 2017-03-15 18:05:03 +08:00
Andrew Wang
cc1292e73a HDFS-9705. Refine the behaviour of getFileChecksum when length = 0. Contributed by Kai Zheng and SammiChen. 2017-03-14 16:41:10 -07:00
Andrew Wang
7515e75103 HDFS-11505. Do not enable any erasure coding policies by default. Contributed by Manoj Govindassamy. 2017-03-14 11:47:25 -07:00
Yiqun Lin
023b941e3b HDFS-11526. Fix confusing block recovery message. Contributed by Yiqun Lin. 2017-03-14 17:49:48 +08:00
Chen Liang
9832ae0ed8 HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-03-13 17:30:10 -07:00
Jing Zhao
55796a0946 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. 2017-03-13 14:24:51 -07:00
Lei Xu
6d356b6b4d HDFS-11340. DataNode reconfigure for disks doesn't remove the failed volumes. (Manoj Govindassamy via lei) 2017-03-10 14:37:13 -08:00
Andrew Wang
819808a016 HDFS-11506. Move ErasureCodingPolicyManager#getSystemDefaultPolicy to test code. Contributed by Manoj Govindassamy. 2017-03-09 17:29:11 -08:00
Masatake Iwasaki
385d2cb777 HDFS-11499. Decommissioning stuck because of failing recovery. Contributed by Lukas Majercak and Manoj Govindassamy. 2017-03-09 21:13:50 +09:00
Andrew Wang
33a38a5341 HDFS-11314. Enforce set of enabled EC policies on the NameNode. 2017-03-08 16:41:44 -08:00
Andrew Wang
5ca6ef0c26 HDFS-10983. OIV tool should make an EC file explicit. Contributed by Manoj Govindassamy. 2017-03-08 15:36:19 -08:00
Andrew Wang
5addacb1e3 HDFS-11152. Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors. Contributed by SammiChen. 2017-03-08 08:47:38 -08:00
Arpit Agarwal
959940b0ab HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-07 10:12:35 -08:00
Arpit Agarwal
b5adc5c301 HDFS-10838. Last full block report received time for each DN should be easily discoverable. Contributed by Surendra Singh Lilhore. 2017-03-06 16:39:53 -08:00
Andrew Wang
5e74196ede HDFS-11498. Make RestCsrfPreventionHandler and WebHdfsHandler compatible with Netty 4.0. 2017-03-06 15:04:13 -08:00
Mingliang Liu
0f336bab9c HDFS-11494. Log message when DN is not selected for block replication. Contributed by Yiqun Lin 2017-03-03 18:47:13 -08:00
Jing Zhao
ac5ae0065a HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou. 2017-03-03 13:31:20 -08:00
Andrew Wang
3085a60430 HDFS-8112. Relax permission checking for EC related operations. 2017-03-03 13:00:22 -08:00
Rakesh Radhakrishnan
3749152b66 HDFS-11416. Refactor out system default erasure coding policy. Contributed by Andrew Wang. 2017-03-03 09:28:03 +05:30
Andrew Wang
8f4817f2c5 Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru." 
This reverts commit e61491d476.
 2017-03-02 16:20:48 -08:00
Arpit Agarwal
e61491d476 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:25 -08:00
Arpit Agarwal
b3ec531f40 HDFS-11461. DataNode Disk Outlier Detection. Contributed by Hanisha Koneru. 2017-03-02 12:45:48 -08:00
Arpit Agarwal
eeca8b0c4e HDFS-11419. HDFS specific network topology classes with storage type info included. Contributed by Chen Liang. 2017-03-02 09:21:27 -08:00
Ming Ma
25c84d279b HDFS-11412. Maintenance minimum replication config value allowable range should be [0, DefaultReplication]. (Manoj Govindassamy via mingma) 2017-03-01 20:23:52 -08:00
Rakesh Radhakrishnan
82ef9accaf HDFS-11428. Change setErasureCodingPolicy to take a required string EC policy name. Contributed by Andrew Wang. 2017-03-01 16:06:02 +05:30
Brahma Reddy Battula
dcd03df9f9 HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula. 2017-03-01 10:45:56 +05:30
Zhe Zhang
d269b488a7 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang. 2017-02-28 09:00:49 -08:00
Andrew Wang
55c07bbed2 HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage. Contributed by Manoj Govindassamy. 2017-02-27 17:07:45 -08:00
Wei-Chiu Chuang
05391c1845 HDFS-10506. OIV's ReverseXML processor cannot reconstruct some snapshot details. Contributed by Akira Ajisaka. 2017-02-25 14:38:50 -08:00
Xiao Chen
e24ed47d9a HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-02-24 16:49:46 -08:00
Arpit Agarwal
d2b3ba9b8f HDFS-11295. Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete(). Contributed by Marton Elek. 2017-02-24 15:44:11 -08:00
Andrew Wang
132f758e3d HDFS-11426. Refactor EC CLI to be similar to storage policies CLI. 2017-02-23 16:00:11 -08:00
Jing Zhao
13d4bcfe35 HDFS-4025. QJM: Sychronize past log segments to JNs that missed them. Contributed by Hanisha Koneru. 2017-02-22 16:33:38 -08:00
Andrew Wang
d150f061f4 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu. 2017-02-22 15:16:09 -08:00
Andrew Wang
8ef7ebbc71 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu. 2017-02-21 14:54:20 -08:00
Andrew Wang
fc9ad3ce3a HDFS-11405. Rename "erasurecode" CLI subcommand to "ec". Contributed by Manoj Govindassamy. 2017-02-21 13:55:27 -08:00
Vinayakumar B
4804050630 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore) 2017-02-21 18:13:19 +05:30
Xiao Chen
02c549484a HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp. 2017-02-16 18:07:55 -08:00
Wei-Chiu Chuang
5690b51ef7 HDFS-11100. Recursively deleting file protected by sticky bit should fail. Contributed by John Zhuge. 2017-02-16 05:39:37 -08:00
Ming Ma
a136936d01 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma) 2017-02-15 20:24:07 -08:00
Mingliang Liu
0741dd3b9a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore 2017-02-15 11:37:26 -08:00
Wei-Chiu Chuang
859bd159ae HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang. 2017-02-15 02:50:35 -08:00
Akira Ajisaka
8acb376c9c HDFS-11238. Fix checkstyle warnings in NameNode#createNameNode. Contributed by Ethan Li. 2017-02-15 16:53:50 +09:00
Yongjun Zhang
8e53f2b9b0 HDFS-11391. Numeric usernames do no work with WebHDFS FS write access. (Pierre Villard via Yongjun Zhang) 2017-02-14 13:40:53 -08:00
Yiqun Lin
4164a2032a HDFS-11408. The config name of balance bandwidth is out of date. Contributed by Yiqun Lin. 2017-02-14 18:57:20 +08:00
Chris Douglas
4ed33e9ca3 HDFS-11026. Convert BlockTokenIdentifier to use Protobuf. Contributed by Ewan Higgs. 2017-02-13 11:29:18 -08:00
Yiqun Lin
663e683adf HDFS-10219. Change the default value for dfs.namenode.reconstruction.pending.timeout-sec from -1 to 300. Contributed by Yiqun Lin. 2017-02-06 19:19:30 +08:00
Yiqun Lin
9cbbd1eae8 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 2017-02-06 13:15:16 +08:00
Arpit Agarwal
6aa09dc28a HDFS-11112. Journal Nodes should refuse to format non-empty directories. Contributed by Yiqun Lin. 2017-02-01 16:51:58 -08:00
Jing Zhao
b6f290d5b6 HDFS-11370. Optimize NamenodeFsck#getReplicaInfo. Contributed Takanobu Asanuma. 2017-02-01 11:21:35 -08:00
Lei Xu
bec9b7aa1d HDFS-11335. Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf. (Manoj Govindassamy via lei) 2017-02-01 14:42:51 +08:00
Xiao Chen
88da9f6b67 Revert "HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang." 
This reverts commit ada233b7cd.
 2017-01-26 13:42:50 -08:00
Arpit Agarwal
7c1cc30b3c HDFS-11369. Change exception message in StorageLocationChecker. 2017-01-26 09:12:38 -08:00
Jing Zhao
b782bf2156 HDFS-11124. Report blockIds of internal blocks for EC files in Fsck. Contributed by Takanobu Asanuma. 2017-01-25 11:16:17 -08:00
Zhe Zhang
18e1d68209 HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki. 2017-01-25 09:58:39 -08:00
Arpit Agarwal
b57368b6f8 HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 16:58:20 -08:00
Wei-Chiu Chuang
355b907c09 HDFS-11121. Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping. Contributed by Takanobu Asanuma. 2017-01-21 05:56:22 +08:00
Lei Xu
78ae2aed8f HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-19 16:28:31 +08:00
Andrew Wang
a8f1c7f542 HDFS-110268. Correctly reconstruct erasure coding file from FSImage. Contributed by SammiChen. 2017-01-19 00:24:11 -08:00
Lei Xu
63320d1daa HDFS-11287. Storage class member storageDirs should be private to avoid unprotected access by derived classes. (Manoj Govindassamy via lei) 2017-01-19 15:38:20 +08:00
Allen Wittenauer
0eb4b513b7 HADOOP-13673. Update scripts to be smarter when running with privilege 
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
 2017-01-18 14:39:05 -08:00
Andrew Wang
a2a5d7b5bc HDFS-10759. Change fsimage bool isStriped from boolean to an enum. Contributed by Ewan Higgs. 2017-01-18 13:31:05 -08:00
Vinitha Reddy Gankidi
8a0fa0f7e8 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:46:32 -08:00
Akira Ajisaka
9130af3f74 HADOOP-13955. Replace deprecated HttpServer2 and SSLFactory constants. Contributed by John Zhuge. 2017-01-18 18:11:33 +09:00
Xiaoyu Yao
b1fce2b8b1 HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. Contributed by Xiaoyu Yao. 2017-01-17 14:33:26 -08:00
Arpit Agarwal
89bb05d92b HDFS-11339. Fix import. 2017-01-16 23:05:28 -08:00
Arpit Agarwal
79e939d0b8 HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. Contributed by Hanisha Koneru. 2017-01-16 14:43:29 -08:00
Arpit Agarwal
a853b4e1b5 HDFS-11342. Fix FileInputStream leak in loadLastPartialChunkChecksum. Contributed by Chen Liang. 2017-01-16 13:45:22 -08:00
Lei Xu
2f8e9b7e4b Revert "HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei)" 
This reverts commit c18590fce2.
 2017-01-16 15:11:53 +08:00
Anu Engineer
2604e82eba HDFS-11344. The default value of the setting dfs.disk.balancer.block.tolerance.percent is different. Contributed by Yiqun Lin. 2017-01-13 22:38:46 -08:00
Wei-Chiu Chuang
1cde954a4f HDFS-11306. Print remaining edit logs from buffer if edit log can't be rolled. Contributed by Wei-Chiu Chuang. 2017-01-13 11:46:30 -08:00
Brahma Reddy Battula
a5ec1e31a3 HDFS-9935. Remove LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD and unused import from HdfsServerConstants. Contributed Yiqun Lin. 2017-01-11 19:34:01 +05:30
Ming Ma
467f5f1735 HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj Govindassamy via mingma) 2017-01-10 20:12:42 -08:00
Andrew Wang
e69231658d HDFS-11072. Add ability to unset and change directory EC policy. Contributed by Sammi Chen. 2017-01-10 11:32:48 -08:00
Arpit Agarwal
4046794a53 HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-10 10:43:11 -08:00
Lei Xu
c18590fce2 HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-10 19:37:58 +08:00
Jing Zhao
7ec609b289 HDFS-11273. Move TransferFsImage#doGetUrl function to a Util class. Contributed by Hanisha Koneru. 2017-01-09 18:05:33 -08:00
Yongjun Zhang
603cbcd513 HDFS-11292. log lastWrittenTxId etc info in logSyncAll. Contributed by Yongjun Zhang. 2017-01-09 15:44:42 -08:00
Arpit Agarwal
91bf504440 HDFS-11301. Double wrapping over RandomAccessFile in LocalReplicaInPipeline#createStreams. Contributed by Hanisha Koneru. 2017-01-09 10:32:18 -08:00
Xiaoyu Yao
87bb1c49bb HDFS-11279. Cleanup unused DataNode#checkDiskErrorAsync(). Contributed by Hanisha Koneru 2017-01-03 18:25:46 -08:00
Andrew Wang
7fcc73fc0d HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 2017-01-03 09:58:00 -08:00
Xiao Chen
e769957556 HDFS-11275. Check groupEntryIndex and throw a helpful exception on failures when removing ACL. 2016-12-29 10:36:50 -08:00
Lei Xu
a4f66655ec HDFS-11267. Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei) 2016-12-29 16:57:40 +08:00
Lei Xu
e9f1396834 HDFS-11251. ConcurrentModificationException during DataNode#refreshVolumes. (Manoj Govindassamy via lei) 2016-12-29 15:11:25 +08:00
Xiaoyu Yao
603f3ef138 HDFS-11274. Datanode should only check the failed volume upon IO errors. Contributed by Xiaoyu Yao. 2016-12-28 22:08:13 -08:00
Daniel Templeton
e216e8e233 HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-27 13:03:16 -08:00
Mingliang Liu
c0e0ef2969 HDFS-11270. Document the missing options of NameNode bootstrap command. Contributed by Yiqun Lin 2016-12-27 11:21:10 -08:00
Akira Ajisaka
8f218ea284 HDFS-11250. Fix a typo in ReplicaUnderRecovery#setRecoveryID. Contributed by Yiqun Lin. 2016-12-26 16:51:29 +09:00
Akira Ajisaka
483cd06ad4 HDFS-11271. Typo in NameNode UI. Contributed by Wei-Chiu Chuang. 2016-12-26 16:36:54 +09:00
Xiaoyu Yao
4e9029653d HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2016-12-22 23:46:58 -08:00
Kai Zheng
56a13a6a59 HDFS-11216. Add remoteBytesRead counter metrics for erasure coding reconstruction task. Contributed by Sammi Chen 2016-12-22 14:18:54 +08:00
Haohui Mai
f6e80acd68 HDFS-11258. File mtime change could not save to editlog. Contributed by Jimmy Xiang. 2016-12-21 13:04:03 -08:00
Xiaoyu Yao
f678080dbd HDFS-11182. Update DataNode to use DatasetVolumeChecker. Contributed by Arpit Agarwal. 2016-12-20 13:53:32 -08:00
Xiaoyu Yao
5daa8d8631 HDFS-10913. Introduce fault injectors to simulate slow mirrors. Contributed by Xiaobing Zhou. 2016-12-20 13:17:52 -08:00
Xiao Chen
5b7acdd206 HDFS-11195. Return error when appending files by webhdfs rest api fails. Contributed by Yuanbo Liu. 2016-12-20 12:24:00 -08:00
Brahma Reddy Battula
1b401f6a73 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage compression. Contributed by Brahma Reddy Battula. 2016-12-20 20:54:03 +05:30
Xiaoyu Yao
fe4ff64a4a HDFS-10959. Adding per disk IO statistics and metrics in DataNode. Contributed by Xiaoyu Yao. 2016-12-19 22:42:54 -08:00
Akira Ajisaka
d2026ae9e6 HDFS-11262. Remove unused variables in FSImage.java. Contributed by Jagadesh Kiran N. 2016-12-20 14:34:35 +09:00
Anu Engineer
a95639068c HDFS-9911. TestDataNodeLifeline Fails intermittently. Contributed by Yiqun Lin 2016-12-16 09:46:21 -08:00
Xiao Chen
cee0c468b0 HDFS-11188. Change min supported DN and NN versions back to 2.x. Contributed by Andrew Wang. 2016-12-15 16:48:50 -08:00
Wei-Chiu Chuang
aebb9127ba HDFS-11160. VolumeScanner reports write-in-progress replicas as corrupt incorrectly. Contributed by Wei-Chiu Chuang and Yongjun Zhang. 2016-12-15 16:32:50 -08:00
Mingliang Liu
6fce191549 HDFS-11253. FileInputStream leak on failure path in BlockSender. Contributed by Arpit Agarwal 2016-12-15 15:39:15 -08:00
Mingliang Liu
ac0d402c6a HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-15 14:53:53 -08:00
Akira Ajisaka
36947f79ba HDFS-11249. Redundant toString() in DFSConfigKeys.java. Contributed by Jagadesh Kiran N. 2016-12-16 00:33:27 +09:00
Arpit Agarwal
6ba9587d37 HDFS-10958. Add instrumentation hooks around Datanode disk IO. 2016-12-14 11:18:58 -08:00
Kai Zheng
1f14f6d038 HDFS-8411. Add bytes count metrics to datanode for ECWorker. Contributed by Sammi Chen and Andrew Wang 2016-12-14 14:50:50 +08:00
Uma Maheswara Rao G
e24a923db5 HDFS-11164: Mover should avoid unnecessary retries if the block is pinned. Contributed by Rakesh R 2016-12-13 17:09:58 -08:00
Mingliang Liu
754f15bae6 HDFS-11226. cacheadmin, cryptoadmin and storagepolicyadmin should support generic options. Contributed by Brahma Reddy Battula 2016-12-12 18:21:15 -08:00
Akira Ajisaka
2d4731c067 HDFS-11233. Fix javac warnings related to the deprecated APIs after upgrading Jackson. Contributed by Yiqun Lin. 2016-12-13 11:11:15 +09:00
Wei-Chiu Chuang
2a28e8cf04 HDFS-11229. HDFS-11056 failed to close meta file. Contributed by Wei-Chiu Chuang. 2016-12-09 16:02:42 -08:00
Andrew Wang
5bd7dece92 HDFS-11217. Annotate NameNode and DataNode MXBean interfaces as Private/Stable. Contributed by Jagadesh Kiran N. 2016-12-09 12:05:42 -08:00
Vinayakumar B
d1d4aba71b HDFS-11224. Lifeline message should be ignored for dead nodes (Contributed by Vinayakumar B) 2016-12-09 14:53:50 +05:30
Kihwal Lee
13d8e552d2 Revert "HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from DatanodeProtocolProtos. Contributed by Tao Luo." 
This reverts commit 4c8db60092.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
 2016-12-08 15:41:49 -06:00
Kihwal Lee
9f8344db50 Revert "HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger" 
This reverts commit 8c4680852b.
 2016-12-08 15:36:08 -06:00
Xiao Chen
401c731872 HDFS-11197. Listing encryption zones fails when deleting a EZ that is on a snapshotted directory. Contributed by Wellington Chevreuil. 2016-12-08 12:40:20 -08:00
Akira Ajisaka
9ef89ede2f HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin. 2016-12-08 13:38:02 +09:00
Akira Ajisaka
deb4743b2c Revert "HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin." 
This reverts commit 0857641f62.
 2016-12-08 13:37:06 +09:00
Akira Ajisaka
0857641f62 HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin. 2016-12-08 13:34:45 +09:00
Andrew Wang
ea2895f4ed HDFS-8630. WebHDFS : Support get/set/unset StoragePolicy. Contributed by Surendra Singh Lilhore. 2016-12-07 15:52:16 -08:00
Ming Ma
c73e08a6da HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:26:09 -08:00
Andrew Wang
f53b481ffe HDFS-11172. Support an erasure coding policy using RS 10 + 4. Contributed by Wei Zhou. 2016-12-06 12:49:38 -08:00
Xiaoyu Yao
df983b524a HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2016-12-06 11:05:47 -08:00
Mingliang Liu
43cb1678cc HDFS-11211. Add a time unit to the DataNode client trace format. Contributed by Jagadesh Kiran N 2016-12-06 09:12:11 -08:00
Kihwal Lee
4dd4f3a4bb HDFS-11198. NN UI should link DN web address using hostnames. Contributed by Weiwei Yang. 2016-12-06 08:06:50 -06:00
Andrew Wang
08a7253bc0 Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang" 
This reverts commit c7ff34f8dc.
 2016-12-05 23:09:35 -08:00
Andrew Wang
8e63fa98ea HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are decomissioning. Contributed by Weiwei Yang. 2016-12-05 18:13:53 -08:00
Xiaoyu Yao
dcedb72af4 Revert "HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao." 
This reverts commit aeecfa24f4.
 2016-12-05 12:44:20 -08:00
Mingliang Liu
8c4680852b HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-05 11:48:58 -08:00
Wei-Chiu Chuang
1b5cceaffb Revert "HDFS-11201. Spelling errors in the logging, help, assertions and exception messages. Contributed by Grant Sohn." 
This reverts commit b9522e86a5.
 2016-12-05 10:54:43 -08:00
Wei-Chiu Chuang
b9522e86a5 HDFS-11201. Spelling errors in the logging, help, assertions and exception messages. Contributed by Grant Sohn. 2016-12-05 10:48:25 -08:00
Mingliang Liu
c7ff34f8dc HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang 2016-12-02 11:10:13 -08:00
Kihwal Lee
96c574927a HDFS-8674. Improve performance of postponed block scans. Contributed by Daryn Sharp. 2016-12-01 12:11:27 -06:00
Akira Ajisaka
e0fa49234f HDFS-11180. Intermittent deadlock in NameNode when failover happens. 2016-12-01 23:08:59 +09:00
Andrew Wang
7226a71b1f HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers and Andrew Wang. 2016-11-30 15:58:31 -08:00
Ravi Prakash
625df87c7b HDFS-8678. Bring back the feature to view chunks of files in the HDFS file browser. Contributed by Ivo Udelsmann. 2016-11-30 09:12:15 -08:00
Kai Zheng
51e6c1cc3f HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen 2016-11-30 15:52:56 +08:00
Kai Zheng
cfd8076f81 Revert due to an error "HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen" 
This reverts commit 5614f847b2ef2a5b70bd9a06edc4eba06174c6.
 2016-11-30 15:44:52 +08:00
Xiaoyu Yao
aeecfa24f4 HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2016-11-29 20:52:36 -08:00
Arpit Agarwal
eaaa32950c HDFS-11149. Support for parallel checking of FsVolumes. 2016-11-29 20:31:02 -08:00
Andrew Wang
b407d53195 Revert "HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers." 
This reverts commit 09451252fa.
 2016-11-29 10:50:10 -08:00
Andrew Wang
09451252fa HDFS-5517. Lower the default maximum number of blocks per file. Contributed by Aaron T. Myers. 2016-11-29 00:12:35 -08:00
Robert Kanter
47ca9e26fb HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via rkanter) 2016-11-28 18:08:09 -08:00
Kai Zheng
5d5614f847 HDFS-10994. Support an XOR policy XOR-2-1-64k in HDFS. Contributed by Sammi Chen 2016-11-28 14:34:44 +08:00
Andrew Wang
de4894936a HDFS-11368. Erasure Coding: Deprecate replication-related config keys. Contributed by Rakesh R. 2016-11-23 16:42:06 -08:00
Arpit Agarwal
613b902b98 HDFS-11148. Update DataNode to use StorageLocationChecker at startup. 2016-11-22 10:50:25 -08:00
Brahma Reddy Battula
afcf8d38e7 HADOOP-13646. Remove outdated overview.html. Contributed By Brahma Reddy Battula. 2016-11-22 19:45:45 +05:30
Kihwal Lee
49a09179e3 HDFS-10966. Enhance Dispatcher logic on deciding when to give up a source DataNode. Contributed by Mark Wagner and Zhe Zhang. 2016-11-21 10:13:24 -06:00
Akira Ajisaka
d232625f73 HDFS-11113. Document dfs.client.read.striped configuration in hdfs-default.xml. Contributed by Rakesh R. 2016-11-21 14:05:15 +09:00
Arpit Agarwal
04a024b683 HDFS-11147. Remove confusing log output in FsDatasetImpl#getInitialVolumeFailureInfos. Contributed by Chen Liang. 2016-11-16 14:47:34 -08:00
Zhe Zhang
ff0b99eafe HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 11:05:29 -08:00
Arpit Agarwal
3d26717777 HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup. 
This closes #155.
 2016-11-11 15:02:52 -08:00
Xiaoyu Yao
4484b48498 HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. Contributed by Chen Liang. 2016-11-11 14:31:23 -08:00
Brahma Reddy Battula
1ae57f0f75 HDFS-11128. CreateEditsLog throws NullPointerException. Contributed by Hanisha Koneru. 2016-11-11 20:08:15 +05:30
Vinayakumar B
ca68f9cb5b HDFS-9337. Validate required params for WebHDFS requests (Contributed by Jagadesh Kiran N) 2016-11-10 16:51:33 +05:30
Wei-Chiu Chuang
c619e9b43f HDFS-11056. Concurrent append and read operations lead to checksum error. Contributed by Wei-Chiu Chuang. 2016-11-09 09:16:50 -08:00
Brahma Reddy Battula
ed0bebabaa HDFS-9482. Replace DatanodeInfo constructors with a builder pattern. Contributed by Brahma Reddy Battula. 2016-11-08 18:17:07 -08:00
Arpit Agarwal
3fff158587 HDFS-11114. Support for running async disk checks in DataNode. 
This closes #153.
 2016-11-07 18:45:53 -08:00
Akira Ajisaka
049e7d27be HDFS-10970. Update jackson from 1.9.13 to 2.x in hadoop-hdfs. 2016-11-07 11:16:31 +09:00
Xiao Chen
d8bab3dcb6 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu. 2016-11-04 18:06:55 -07:00
Vinayakumar B
69dd5fa2d4 HDFS-11098. Datanode in tests cannot start in Windows after HDFS-10638 (Contributed by Vinayakumar B) 2016-11-04 08:31:43 +05:30
Andrew Wang
5cad93d5c7 HADOOP-11088. Quash unnecessary safemode WARN message during NameNode startup. Contributed by Yiqun Lin. 2016-11-03 15:31:07 -07:00
Arpit Agarwal
b71907b2ae HDFS-11097. Fix warnings for deprecated StorageReceivedDeletedBlocks constructor. Contributed by Yiqun Lin. 2016-11-03 10:27:52 -07:00
Anu Engineer
730cb0cff6 HDFS-10566. Submit plan request should throw exception if Datanode is in non-REGULAR status. Contributed by Xiaobing Zhou. 2016-11-02 17:46:56 -07:00
Lei Xu
b59206190e HDFS-10909. De-duplicate code in ErasureCodingWorker#initializeStripedReadThreadPool and DFSClient#initThreadsNumForStripedReads. (Manoj Govindassamy via lei) 2016-11-02 16:45:25 -07:00
Mingliang Liu
9b0c17f852 HDFS-11095. BlockManagerSafeMode should respect extension period default config value (30s). Contributed by Mingliang Liu 2016-11-02 14:58:24 -07:00
Andrew Wang
f2800f523d Revert "HADOOP-13717. Normalize daemonization behavior of the diskbalancer with balancer and mover." 
This reverts commit bd7f5911c7.
 2016-11-02 13:46:33 -07:00
Andrew Wang
bd7f5911c7 HADOOP-13717. Normalize daemonization behavior of the diskbalancer with balancer and mover. 2016-11-01 16:48:45 -07:00
Anu Engineer
34173a42d4 HDFS-10633. DiskBalancer : Add the description for the new setting dfs.disk.balancer.plan.threshold.percent in HDFSDiskbalancer.md. Contributed by Yiqun Lin. 2016-11-01 11:14:03 -07:00
Arpit Agarwal
a9d68d2e8e HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:39:30 -07:00
Brahma Reddy Battula
e9c7a97089 HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 2016-10-31 00:22:51 +05:30
Mingliang Liu
8a9388e5f6 HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger 2016-10-28 10:53:14 -07:00
Xiao Chen
57187fdb93 HDFS-11064. Mention the default NN rpc ports in hdfs-default.xml. Contributed by Yiqun Lin. 2016-10-27 18:13:06 -07:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Mingliang Liu
9e03ee5279 HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 2016-10-27 16:00:27 -07:00
Erik Krogen
f3ac1f41b8 HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen. 2016-10-27 15:58:25 -07:00
Kihwal Lee
ae48c496dc HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee 2016-10-27 14:17:16 -05:00
Kihwal Lee
db41965996 Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu" 
This reverts commit ac35ee9393.
 2016-10-27 14:07:28 -05:00
Brahma Reddy Battula
ac35ee9393 HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu 2016-10-27 20:20:56 +05:30
Vinayakumar B
1cf6ec4ad4 HDFS-8492. DN should notify NN when client requests a missing block (Contributed by Walter Su) 2016-10-27 16:44:00 +05:30
Vinayakumar B
77f2684dd3 HDFS-10769. BlockIdManager.clear doesn't reset the counter for blockGroupIdGenerator (Contributed by Yiqun Lin and Rakesh R) 2016-10-27 14:16:03 +05:30
Akira Ajisaka
9c680d0b05 HDFS-11049. The description of dfs.block.replicator.classname is not clear. Contributed by Yiqun Lin. 2016-10-27 16:30:57 +09:00
Vinayakumar B
a895c66255 HDFS-9929. Duplicate keys in NAMENODE_SPECIFIC_KEYS (Contributed by Akira Ajisaka) 2016-10-27 12:20:15 +05:30
Anu Engineer
9f32364d28 HDFS-11038. DiskBalancer: support running multiple commands in single test. Contributed by Xiaobing Zhou. 2016-10-26 19:39:02 -07:00
Lei Xu
f209e93566 HDFS-10638. Modifications to remove the assumption that StorageLocation is associated with java.io.File in Datanode. (Virajith Jalaparti via lei) 2016-10-26 10:32:35 -07:00
Kihwal Lee
e90af4a89b HDFS-11053. Unnecessary superuser check in versionRequest(). Contributed by Kihwal Lee. 2016-10-26 09:16:13 -05:00
Kai Zheng
287effff93 HDFS-10935. TestFileChecksum fails in some cases. Contributed by Sammi Chen 2016-10-26 09:40:31 +06:00
Arpit Agarwal
9dce9011ad HDFS-11054. Suppress verbose log message in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2016-10-25 19:50:56 -07:00
Zhe Zhang
f6367c5f44 HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee. 2016-10-25 10:19:13 -07:00
Akira Ajisaka
dc3272bfe8 HDFS-11046. Duplicate '-' in the daemon log name. 2016-10-25 10:07:43 +09:00
Kihwal Lee
9d175853b0 HDFS-10997. Reduce number of path resolving methods. Contributed by Daryn Sharp. 2016-10-24 17:14:51 -05:00