Commit Graph

3085 Commits

Author SHA1 Message Date
Uma Maheswara Rao G
3a4ff7776e HDFS-9694. Make existing DFSClient#getFileChecksum() work for striped blocks. Contributed by Kai Zheng 2016-03-26 19:58:09 -07:00
Lei Xu
fde8ac5d85 Add missing files from HDFS-9005. (lei) 2016-03-25 17:11:35 -07:00
Lei Xu
4fcfea71bf HDFS-9005. Provide support for upgrade domain script. (Ming Ma via Lei Xu) 2016-03-25 17:09:12 -07:00
Andrew Wang
e3bb38d625 HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed by Xiao Chen. 2016-03-21 11:39:05 -07:00
Sangjin Lee
cd8b6889a7 HDFS-9579. Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level (Ming Ma via sjlee) 2016-03-19 14:02:04 -07:00
Kihwal Lee
63c966a3fb HDFS-9874. Long living DataXceiver threads cause volume shutdown to block. Contributed by Rushabh Shah. 2016-03-18 10:24:59 -05:00
Colin Patrick Mccabe
dc951e606f HDFS-9949. Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared (Harsh J via cmccabe) 2016-03-17 10:37:42 -07:00
Zhe Zhang
32d043d9c5 HDFS-9857. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-1]. Contributed by Rakesh R. 2016-03-16 16:53:58 -07:00
Kihwal Lee
d457401784 HDFS-9904. testCheckpointCancellationDuringUpload occasionally fails. Contributed by Lin Yiqun. 2016-03-15 10:52:47 -05:00
Arpit Agarwal
e01c6ea688 HDFS-1477. Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart. (Contributed by Xiaobing Zhou) 2016-03-10 19:03:55 -08:00
Colin Patrick Mccabe
bd49354c6d HDFS-9934. ReverseXML oiv processor should bail out if the XML file's layoutVersion doesn't match oiv's (cmccabe) 2016-03-10 13:41:06 -08:00
Zhe Zhang
7600e3c48f HDFS-7866. Erasure coding: NameNode manages multiple erasure coding policies. Contributed by Rui Li. 2016-03-08 22:30:30 -08:00
Andrew Wang
a14a6f08ee HDFS-9919. TestStandbyCheckpoints#testNonPrimarySBNUploadFSImage waitForCheckpoint incorrectly. Contributed by Lin Yiqun. 2016-03-08 11:54:50 -08:00
Jing Zhao
743a99f2db HDFS-8786. Erasure coding: use simple replication for internal blocks on decommissioning datanodes. Contributed by Rakesh R. 2016-03-08 10:24:57 -08:00
Masatake Iwasaki
d718fc1ee5 HDFS-9865. TestBlockReplacement fails intermittently in trunk (Lin Yiqun via iwasakims) 2016-03-08 02:07:18 +09:00
Chris Nauroth
2759689d7d HDFS-9239. DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness. Contributed by Chris Nauroth. 2016-03-04 15:29:50 -08:00
Zhe Zhang
3e8099a45a HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 2016-03-04 09:48:05 -08:00
Arpit Agarwal
27941a1811 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-03-02 18:35:28 -08:00
Colin Patrick Mccabe
700b0e4019 HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from an XML file (cmccabe) 2016-03-02 17:56:11 -08:00
Xiaoyu Yao
5abf051249 HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed by Austin Donnelly and Chris Douglas. 2016-03-02 09:23:17 -08:00
Akira Ajisaka
27e0681f28 HDFS-9851. NameNode throws NPE when setPermission is called on a path that does not exist. Contributed by Brahma Reddy Battula. 2016-03-02 14:43:03 +09:00
Akira Ajisaka
e2ddf82469 HDFS-9766. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently. Contributed by Xiao Chen. 2016-03-02 14:17:35 +09:00
Andrew Wang
4abb2fa687 HDFS-9881. DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones. 2016-03-01 16:35:39 -08:00
Kihwal Lee
e76b13c415 HDFS-9880. TestDatanodeRegistration fails occasionally. Contributed by Kihwal Lee. 2016-03-01 17:00:18 -06:00
Kihwal Lee
2c8496ebf3 HDFS-8791. block ID-based DN storage layout can be very slow for datanode on ext4. Contributed by Chris Trezzo. 2016-03-01 15:04:59 -06:00
Jing Zhao
2151716832 HDFS-7964. Add support for async edit logging. Contributed by Daryn Sharp. 2016-02-29 15:34:43 -08:00
Jing Zhao
408f2c807b HDFS-9866. BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance. Contributed by Jing Zhao. 2016-02-28 14:54:49 -08:00
Tsz-Wo Nicholas Sze
d1d4e16690 HDFS-9710. DN can be configured to send block receipt IBRs in batches. 2016-02-26 15:32:25 -08:00
Karthik Kambatla
c2460dad64 HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure clusters. (Daniel Templeton via kasha) 2016-02-25 16:31:09 -08:00
Zhe Zhang
8808779db3 HDFS-9734. Refactoring of checksum failure report related codes. Contributed by Kai Zheng.
Change-Id: Ie69a77e3498a360959f8e213c51fb2b17c28b64a
2016-02-25 09:55:50 -08:00
Zhe Zhang
ccff6035f5 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen.
Change-Id: I54b82e0b11929082057b4d21bfcf285222c94cdb
2016-02-25 09:24:32 -08:00
Tsz-Wo Nicholas Sze
6979cbfc1f HDFS-9838. Refactor the excessReplicateMap to a class. 2016-02-24 19:42:47 -08:00
Jing Zhao
47b92f2b6f HDFS-9837. BlockManager#countNodes should be able to detect duplicated internal blocks. Contributed by Jing Zhao. 2016-02-24 15:13:56 -08:00
Kihwal Lee
d27d7fc72e HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla. 2016-02-24 12:04:01 -06:00
Zhe Zhang
def754ec06 HDFS-9816. Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]. Contributed by Rui Li.
Change-Id: I64b57bab4722cdc6e1e3148c3a3a401370249afe
2016-02-23 14:05:17 -08:00
Zhe Zhang
647a35e996 HDFS-9844. Correct path creation in getTrashRoot to handle root dir.
Change-Id: I5afe6c7176063892e3e632698c849bd9f4b92906
2016-02-23 13:29:39 -08:00
Colin Patrick Mccabe
211c78c090 HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via cmccabe) 2016-02-23 12:01:20 -08:00
Vinayakumar B
342c9572bf HDFS-9425. Expose number of blocks per volume as a metric (Contributed by Brahma Reddy Battula) 2016-02-22 09:29:23 +05:30
Jing Zhao
e54cc29312 HDFS-9818. Correctly handle EC reconstruction work caused by not enough racks. Contributed by Jing Zhao. 2016-02-19 19:02:23 -08:00
Tsz-Wo Nicholas Sze
bbfaf3c271 HADOOP-10865. Add a Crc32 chunked verification benchmark for both directly and non-directly buffer cases. 2016-02-18 11:47:33 -08:00
cnauroth
5d1889a66d HDFS-9711. Integrate CSRF prevention filter in WebHDFS. Contributed by Chris Nauroth. 2016-02-18 10:07:28 -08:00
Andrew Wang
3a23dc683c HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. Contributed by Wei Zhou. 2016-02-17 11:29:58 -08:00
Vinayakumar B
2536ece7b6 HDFS-9787. SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false. (Contributed by Guocui Mi) 2016-02-17 10:41:24 +05:30
Junping Du
77ba5add0d HDFS-9456. BlockPlacementPolicyWithNodeGroup should override verifyBlockPlacement(). Contributed by Xiaobing Zhou. 2016-02-16 18:55:55 -08:00
Akira Ajisaka
8ed07bd3e6 HDFS-9765. TestBlockScanner#testVolumeIteratorWithCaching fails intermittently. (aajisaka) 2016-02-15 19:08:03 +09:00
Akira Ajisaka
b0738ae673 HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. Contributed by Brahma Reddy Battula. 2016-02-15 15:35:06 +09:00
Akira Ajisaka
ac5da11399 HDFS-9691. TestBlockManagerSafeMode#testCheckSafeMode fails intermittently. Contributed by Mingliang Liu. 2016-02-15 14:00:09 +09:00
Arpit Agarwal
1de1641f17 HDFS-9801. ReconfigurableBase should update the cached configuration. (Arpit Agarwal) 2016-02-12 12:41:04 -08:00
Jing Zhao
972782d956 HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. Contributed by Jing Zhao. 2016-02-12 11:07:52 -08:00
Karthik Kambatla
33ce7f6c07 HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. (Daniel Templeton via kasha) 2016-02-11 20:37:14 -08:00
Lei Xu
c7fcec24b8 HDFS-9795. OIV Delimited should show which files are ACL-enabled (lei) 2016-02-11 20:33:51 -08:00
Zhe Zhang
0aa8c82894 HDFS-9755. Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]. Contributed by Rui Li.
Change-Id: I2100bc27ad484f83c9cb2d2e5bb232f4f74fd286
2016-02-11 10:14:09 -08:00
Jing Zhao
19adb2bc64 HDFS-9789. Correctly update DataNode's scheduled block size when writing small EC file. Contributed by Jing Zhao. 2016-02-10 21:24:00 -08:00
Karthik Kambatla
bc425a623f HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) 2016-02-10 10:08:05 -08:00
Akira Ajisaka
802a7edfae HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-10 23:49:14 +09:00
Akira Ajisaka
5e565ce7a7 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta."
This reverts commit fe124da5ff.
2016-02-10 23:43:33 +09:00
Uma Maheswara Rao G
a7fce9ab41 HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value. Contributed by Kuhu Shukla. 2016-02-09 15:14:30 -08:00
Zhe Zhang
a0fb2eff9b HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork. Contributed by Rakesh R.
Change-Id: I6dfc8efd94fa2bbb4eec0e4730a5a4f92c8a5519
2016-02-09 14:43:59 -08:00
Allen Wittenauer
401ae4ecdb HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler (Ryan Sasson via aw) 2016-02-09 14:15:21 -08:00
zhezhang
dbe49c1bd6 HDFS-9244. Support nested encryption zones.
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
2016-02-08 16:30:51 -08:00
Kihwal Lee
193d27de0a HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su. 2016-02-08 12:16:29 -06:00
Akira Ajisaka
fe124da5ff HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-06 19:31:01 +09:00
Tsz-Wo Nicholas Sze
4e5e1c0f99 HDFS-9726. Refactor IBR code to a new class. 2016-02-05 23:17:59 +08:00
Konstantin V Shvachko
81ef1a1f55 HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of -namenode. Contributed by Mingliang Liu. 2016-02-03 14:59:46 -08:00
zhezhang
a0f5e83f28 HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping related tests. Contributed by Rui Li.
Change-Id: I5b02f5bde4d343b7529c6a7fe5df73bd99c6cb24
2016-02-03 14:41:07 -08:00
Jing Zhao
7badf15604 HDFS-9748. Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice. Contributed by Walter Su. 2016-02-03 13:30:09 -08:00
Vinayakumar B
eb2fb943fd HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() (Contributed by Chang Li) 2016-02-03 11:14:29 +05:30
Vinayakumar B
04375756a5 HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed by Lei (Eddy) Xu) 2016-02-03 07:35:01 +05:30
zhezhang
6d1213860f HDFS-9403. Erasure coding: some EC tests are missing timeout. Contributed by Gao Rui.
Change-Id: I13c4bc0e566c4f8af5371d26f8e18a749ed41d8b
2016-02-02 14:03:45 -08:00
zhezhang
4ae543fdcd HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to BlockECReconstructionCommand. Contributed by Rakesh R.
Change-Id: I405365a8395770e494b92bfe9651f4f0366d8f28
2016-02-02 12:32:08 -08:00
Colin Patrick Mccabe
dd9ebf6eed HDFS-9260. Improve the performance and GC friendliness of NameNode startup and full block reports (Staffan Friberg via cmccabe) 2016-02-02 11:23:00 -08:00
Lei Xu
9d494f0c0e HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference. (Xiao Chen via lei) 2016-02-02 10:42:01 -08:00
Vinayakumar B
4d7055e0c8 HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys before initialize (Contributed by DENG FEI) 2016-02-02 12:51:35 +05:30
Yongjun Zhang
34ab50ea92 HDFS-9406. FSImage may get corrupted after deleting snapshot. (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang) 2016-02-01 13:56:55 -08:00
Lei Xu
e50aa53eed HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei) 2016-02-01 12:56:53 -08:00
Steve Loughran
af2dccbca5 HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames (Mingliang Liu via stevel) 2016-02-01 16:01:50 +00:00
Colin Patrick Mccabe
f67149ab08 HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had an error (Rakesh R via cmccabe) 2016-01-28 19:54:50 -08:00
Tsz-Wo Nicholas Sze
662e17b46a HDFS-9654. Code refactoring for HDFS-8578. 2016-01-28 10:58:03 +08:00
Jing Zhao
ec25c7f9c7 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 16:34:40 -08:00
Jing Zhao
3a9571308e Revert "HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu."
This reverts commit 8a91109d16.
2016-01-27 16:31:19 -08:00
Jing Zhao
8a91109d16 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:48:47 -08:00
Tsz-Wo Nicholas Sze
45c763ad61 HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. 2016-01-26 11:20:13 +08:00
Tsz-Wo Nicholas Sze
bd909ed9f2 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-26 10:32:51 +08:00
Jitendra Pandey
e8650fea1f HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. Contributed by Mingliang Liu. 2016-01-25 16:08:46 -08:00
Arpit Agarwal
d62b4a4de7 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:17:05 -08:00
Eric Yang
10a2bc0dff HDFS-9653. Added blocks pending deletion report to dfsadmin.
(Weiwei Yang via eyang)
2016-01-24 14:19:49 -08:00
Allen Wittenauer
d22c4239a4 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2016-01-22 12:15:22 -08:00
Jing Zhao
95363bcc7d HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block. Contributed by Jing Zhao. 2016-01-22 09:46:02 -08:00
Kihwal Lee
2fd19b9674 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:10:06 -06:00
Masatake Iwasaki
b2ffcc2915 HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block (iwasakims) 2016-01-22 12:28:38 +09:00
Jing Zhao
c304890c8c HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao. 2016-01-21 11:13:17 -08:00
Kihwal Lee
3616c7b855 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne. 2016-01-21 09:32:57 -06:00
Vinayakumar B
b7372b7166 HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI) 2016-01-21 19:17:05 +05:30
Yongjun Zhang
57d0a94305 HDFS-6054. MiniQJMHACluster should not use static port to avoid binding failure in unit test. (Yongjun Zhang) 2016-01-19 22:54:47 -08:00
Andrew Wang
c07f7fa8ff HDFS-9624. DataNode start slowly due to the initial DU command operations. (Lin Yiqun via wang) 2016-01-15 11:28:46 -08:00
Yongjun Zhang
817cc1f02a HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message change. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-14 12:36:36 -08:00
Yongjun Zhang
25051c3bd0 HDFS-9569. Log the name of the fsimage being loaded for better supportability. (Yongjun Zhang) 2016-01-12 09:22:22 -08:00
Jing Zhao
9f4bf3bdf9 HDFS-9621. getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory. Contributed by Jing Zhao. 2016-01-11 11:31:59 -08:00
Lei Xu
fd8065a763 HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (Tony Wu via lei) 2016-01-08 15:00:30 -08:00
Zhe Zhang
0af2022e6d HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. Contributed by Xiao Chen.
Change-Id: Ia766a5bfb89f8cf8a48b473bb27c30d5818ebfc0
2016-01-08 13:14:36 -08:00