big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,983 Commits 2 Branches 2 Tags 264 MiB
211c78c090
Commit Graph

2999 Commits

Author SHA1 Message Date
Allen Wittenauer
d22c4239a4 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2016-01-22 12:15:22 -08:00
Jing Zhao
95363bcc7d HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block. Contributed by Jing Zhao. 2016-01-22 09:46:02 -08:00
Kihwal Lee
2fd19b9674 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:10:06 -06:00
Masatake Iwasaki
b2ffcc2915 HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block (iwasakims) 2016-01-22 12:28:38 +09:00
Jing Zhao
c304890c8c HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao. 2016-01-21 11:13:17 -08:00
Kihwal Lee
3616c7b855 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne. 2016-01-21 09:32:57 -06:00
Vinayakumar B
b7372b7166 HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI) 2016-01-21 19:17:05 +05:30
Yongjun Zhang
57d0a94305 HDFS-6054. MiniQJMHACluster should not use static port to avoid binding failure in unit test. (Yongjun Zhang) 2016-01-19 22:54:47 -08:00
Andrew Wang
c07f7fa8ff HDFS-9624. DataNode start slowly due to the initial DU command operations. (Lin Yiqun via wang) 2016-01-15 11:28:46 -08:00
Yongjun Zhang
817cc1f02a HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message change. (Wei-Chiu Chuang via Yongjun Zhang) 2016-01-14 12:36:36 -08:00
Yongjun Zhang
25051c3bd0 HDFS-9569. Log the name of the fsimage being loaded for better supportability. (Yongjun Zhang) 2016-01-12 09:22:22 -08:00
Jing Zhao
9f4bf3bdf9 HDFS-9621. getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory. Contributed by Jing Zhao. 2016-01-11 11:31:59 -08:00
Lei Xu
fd8065a763 HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (Tony Wu via lei) 2016-01-08 15:00:30 -08:00
Zhe Zhang
0af2022e6d HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. Contributed by Xiao Chen. 
Change-Id: Ia766a5bfb89f8cf8a48b473bb27c30d5818ebfc0
 2016-01-08 13:14:36 -08:00
Kihwal Lee
38c4c14472 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee. 2016-01-08 11:13:58 -06:00
Vinayakumar B
6702e7d66a HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the correct namenode (Contributed by Wei-Chiu Chuang) 2016-01-07 14:20:18 +05:30
Vinayakumar B
34cd7cd765 HDFS-9600. do not check replication if the block is under construction (Contributed by Phil Yang) 2016-01-07 11:27:42 +05:30
Arpit Agarwal
67c9780609 HDFS-9498. Move code that tracks blocks with future generation stamps to BlockManagerSafeMode. (Contributed by Mingliang Liu) 2016-01-06 10:30:59 -08:00
cnauroth
84a8147791 HDFS-9376. TestSeveralNameNodes fails occasionally. Contributed by Masatake Iwasaki. 2015-12-29 14:37:03 -08:00
cnauroth
99cf2ecee9 HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. Contributed by Xiao Chen. 2015-12-29 10:56:59 -08:00
Benoy Antony
8cfd672397 HDFS-9597. BaseReplicationPolicyTest should update data node stats after adding a data node. Contributed by Wei-Chiu Chuang. 2015-12-25 08:55:39 -08:00
Benoy Antony
df83230948 HDFS-9034. StorageTypeStats Metric should not count failed storage. Contributed by Surendra Singh Lilhore. 2015-12-22 15:28:17 -08:00
Kihwal Lee
867048c3e4 HDFS-7163. WebHdfsFileSystem should retry reads according to the configured retry policy. Contributed by Eric Payne. 2015-12-22 14:08:23 -06:00
cnauroth
5c0ff69618 HDFS-9458. TestBackupNode always binds to port 50070, which can cause bind failures. Contributed by Xiao Chen. 2015-12-22 10:29:35 -08:00
Colin Patrick Mccabe
bb540ba85a HDFS-9589. Block files which have been hardlinked should be duplicated before the DataNode appends to the them (cmccabe) 2015-12-22 09:36:50 -08:00
Tsz-Wo Nicholas Sze
70d6f20126 HDFS-9575. Use byte array for internal block indices in a striped block. Contributed by jing9 2015-12-22 14:47:57 +08:00
Zhe Zhang
0087734cc1 HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed due to unexpected number of invalidate blocks. Contributed by Wei-Chiu Chuang. 
Change-Id: I9411dfb1dd4605cbcc515ca3361218d35d7c5fe3
 2015-12-21 22:38:30 -08:00
Uma Mahesh
52ad9125b8 HDFS-9582. TestLeaseRecoveryStriped file missing Apache License header and not well formatted. (umamahesh) 2015-12-20 13:10:57 -08:00
Uma Mahesh
0f82b5d878 YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh) 2015-12-19 23:10:13 -08:00
Zhe Zhang
ef3f3f6bb1 HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown() is wrong. Contributed by Wei-Chiu Chuang. 
Change-Id: Idde8522e11c77f82b376e0c90a9bf33d163d5cb9
 2015-12-18 16:04:47 -08:00
Zhe Zhang
61ab0440f7 HDFS-9173. Erasure Coding: Lease recovery for striped file. Contributed by Walter Su and Jing Zhao. 
Change-Id: I51703a61c9d8454f883028f3f6acb5729fde1b15
 2015-12-18 15:57:48 -08:00
Vinayakumar B
bfadf11b36 HDFS-9393. After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy (Contributed by J.Andreina) 2015-12-18 11:38:12 +05:30
cnauroth
6da9deec20 HDFS-9571. Fix ASF Licence warnings in Jenkins reports. Contributed by Brahma Reddy Battula. 2015-12-17 16:26:01 -08:00
Kihwal Lee
5cb1e0118b HDFS-9533. seen_txid in the shared edits directory is modified during bootstrapping. Contributed by Kihwal Lee. 2015-12-17 17:11:16 -06:00
Arpit Agarwal
2f623fb8cc HDFS-9515. NPE when MiniDFSCluster#shutdown is invoked on uninitialized reference. (Contributed by Wei-Chiu Chuang) 2015-12-17 13:57:08 -08:00
Arpit Agarwal
72a3f649e4 HDFS-9565. TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes is flaky due to race condition. (Contributed by Wei-Chiu Chuang) 2015-12-17 10:36:07 -08:00
Uma Mahesh
f741476146 HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh) 2015-12-16 18:16:39 -08:00
Andrew Wang
3c0adac88c HDFS-9300. TestDirectoryScanner.testThrottle() is still a little flakey. Contributed by Daniel Templeton. 2015-12-16 16:21:06 -08:00
cnauroth
c470c8953d HDFS-9557. Reduce object allocation in PB conversion. Contributed by Daryn Sharp. 2015-12-16 11:10:40 -08:00
Lei Xu
de522d2cd4 Revert "Revert "HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe)"" 
This reverts commit 7f393a6f61.
 2015-12-14 11:25:09 -08:00
Lei Xu
f229772f99 HDFS-9281. Change TestDeleteBlockPool to not explicitly use File to check block pool existence. (lei) 2015-12-14 11:02:07 -08:00
Yongjun Zhang
bf5295b118 HDFS-9514. TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; exception being swallowed. (Wei-Chiu Chuang via Yongjun Zhang) 2015-12-11 19:52:38 -08:00
Uma Mahesh
796a676d18 HDFS-9528. Cleanup namenode audit/log/exception messages. (szetszwo via umamahesh) 2015-12-11 17:57:35 -08:00
Allen Wittenauer
576b569b6c Revert "HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw)" 
This reverts commit 832b3cbde1.
 2015-12-11 10:59:42 -08:00
Kihwal Lee
a48301791e HDFS-9445. Datanode may deadlock while handling a bad volume. Contributed by Walter Su. 2015-12-11 08:44:47 -06:00
Uma Mahesh
e363417e7b HDFS-9472. concat() API does not give proper exception messages on ./reserved relative path (Rakesh R via umamahesh) 2015-12-10 23:55:29 -08:00
Lei Xu
7f393a6f61 Revert "HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe)" 
This reverts commit a153b9601a.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
 2015-12-10 10:57:33 -10:00
Tsz-Wo Nicholas Sze
132478e805 HDFS-9527. The return type of FSNamesystem.getBlockCollection should be changed to INodeFile. 2015-12-09 17:55:28 -08:00
Allen Wittenauer
832b3cbde1 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2015-12-08 12:56:38 -08:00
Lei Xu
4265a85f6d HDFS-9491. Tests should get the number of pending async deletes via FsDatasetTestUtils. (Tony Wu via lei) 2015-12-04 18:06:16 -08:00
Arpit Agarwal
bc6d8d9814 HDFS-9214. Add missing license header 2015-12-04 15:33:09 -08:00
Arpit Agarwal
9d817fa1b1 HDFS-9214. Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart. (Contributed by Xiaobing Zhou) 2015-12-04 14:46:46 -08:00
Yongjun Zhang
59dbe8b3e9 HDFS-9474. TestPipelinesFailover should not fail when printing debug message. (John Zhuge via Yongjun Zhang) 2015-12-04 13:45:01 -08:00
Colin P. McCabe
e02bbeb886 HDFS-9267. TestDiskError should get stored replicas through FsDatasetTestUtils. (Lei (Eddy) Xu via Colin P. McCabe) 2015-12-04 12:51:19 -08:00
Xiaoyu Yao
cbc7b6bf97 HDFS-8831. Trash Support for deletion in HDFS encryption zone. Contributed by Xiaoyu Yao. 2015-12-04 10:39:45 -08:00
Lei Xu
0ac8fb4b33 HDFS-9490. MiniDFSCluster should change block generation stamp via FsDatasetTestUtils. (Tony Wu via lei) 2015-12-04 10:24:55 -08:00
Konstantin V Shvachko
924a33d02d HDFS-9484. NNThroughputBenchmark$BlockReportStats should not send empty block reports. Contributed by Mingliang Liu. 2015-12-03 15:03:34 -08:00
Konstantin V Shvachko
e71aa717d4 HDFS-9436. Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default. Contributed by Mingliang. 2015-12-03 15:03:33 -08:00
Colin Patrick Mccabe
53e3bf7e70 HDFS-9429. Tests in TestDFSAdminWithHA intermittently fail with EOFException (Xiao Chen via Colin P. McCabe) 2015-12-01 23:21:21 -08:00
Jing Zhao
a49cc74b4c HDFS-9129. Move the safemode block count into BlockManager. Contributed by Mingliang Liu. 2015-12-01 16:09:19 -08:00
Arpit Agarwal
58f6f54eea HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails intermittently. (Contributed by Wei-Chiu Chuang) 2015-12-01 13:32:32 -08:00
Akira Ajisaka
1c05393b51 HDFS-9336. deleteSnapshot throws NPE when snapshotname is null. Contributed by Brahma Reddy Battula. 2015-12-01 11:29:09 +08:00
Andrew Wang
9b8e50b424 HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang. 2015-11-30 14:32:51 -08:00
Konstantin V Shvachko
fc799ab16c HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by Brahma Reddy Battula. 2015-11-25 13:52:09 -08:00
Xiaoyu Yao
e3d673901b HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info. Contributed by Xiaoyu Yao. 2015-11-25 13:41:06 -08:00
Haohui Mai
b21dffb1fe HDFS-9451. Clean up depreated umasks and related unit tests. Contributed by Wei-Chiu Chuang. 2015-11-25 12:47:24 -08:00
Yongjun Zhang
8176ea7dc6 HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. (John Zhuge via Yongjun Zhang) 2015-11-25 07:40:16 -08:00
Tsz-Wo Nicholas Sze
78ec38b2ed HDFS-8807. dfs.datanode.data.dir does not handle spaces between storageType and URI correctly. Contributed by Anu Engineer 2015-11-24 16:01:55 -08:00
Xiaoyu Yao
fe5624b85d HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou. 2015-11-24 12:47:57 -08:00
Ming Ma
0e54b164a8 HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess replicas. (Xiao Chen via mingma) 2015-11-24 10:30:24 -08:00
cnauroth
1777608fa0 HDFS-6101. TestReplaceDatanodeOnFailure fails occasionally. Contributed by Wei-Chiu Chuang. 2015-11-24 09:39:21 -08:00
Uma Mahesh
411e2b2e7c HDFS-9433. DFS getEZForPath API on a non-existent file should throw FileNotFoundException (Rakesh R via umamahesh) 2015-11-23 17:54:31 -08:00
Walter Su
201f14e271 HDFS-9423. Fix intermittent failure of TestEditLogTailer. Contributed by Masatake Iwasaki. 2015-11-23 18:49:40 +08:00
Walter Su
001ecf962c HDFS-9435. TestBlockRecovery#testRBWReplicas is failing intermittently. Contributed by Rakesh R. 2015-11-23 15:49:58 +08:00
Walter Su
5aba093361 HDFS-9428. Fix intermittent failure of TestDNFencing.testQueueingWithAppend. Contributed by Masatake Iwasaki. 2015-11-23 15:37:45 +08:00
Uma Mahesh
5b714a2819 HDFS-9348. Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly. (Rakesh R via umamahesh) 2015-11-21 22:38:36 -08:00
Ming Ma
1c4951a7a0 HDFS-8056. Decommissioned dead nodes should continue to be counted as dead after NN restart. (mingma) 2015-11-19 10:04:01 -08:00
Akira Ajisaka
ac1aa6c819 HDFS-9431. DistributedFileSystem#concat fails if the target path is relative. Contributed by Kazuho Fujii. 2015-11-19 20:55:01 +09:00
Colin Patrick Mccabe
dfbde3fc51 HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp. (Lei (Eddy) Xu via cmccabe) 2015-11-17 10:48:15 -08:00
Walter Su
621cbb4f69 HDFS-9358. TestNodeCount#testNodeCount timed out. Contributed by Masatake Iwasaki. 2015-11-17 10:51:29 +08:00
Xiaoyu Yao
a4f62a2d58 HDFS-9421. NNThroughputBenchmark replication test NPE with -namenode option. Contributed by Mingliang Liu. 2015-11-16 16:42:15 -08:00
Xiaoyu Yao
2701f2d255 HDFS-9387. Fix namenodeUri parameter parsing in NNThroughputBenchmark. Contributed by Mingliang Liu. 2015-11-16 16:29:47 -08:00
Ming Ma
02653add98 HDFS-9413. getContentSummary() on standby should throw StandbyException. (Brahma Reddy Battula via mingma) 2015-11-16 09:32:40 -08:00
Walter Su
cccf88480b HDFS-9410. Some tests should always reset sysout and syserr. Contributed by Xiao Chen. 2015-11-13 14:26:43 +08:00
Zhe Zhang
7b00c8e20e HDFS-8968. Erasure coding: a comprehensive I/O throughput benchmark tool. Contributed by Rui Li. 
Change-Id: I01ff5b04727fd79e6373582d4815c5e7b2096c67
 2015-11-12 11:30:39 -08:00
Kihwal Lee
a2a104f3d3 HDFS-9396. Total files and directories on jmx and web UI on standby is uninitialized. Contributed by Kihwal Lee. 2015-11-11 15:24:43 -06:00
Zhe Zhang
73b94d7899 HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. Contributed by Xiao Chen. 
Change-Id: I9e42f724f27924cf73891425a832de37ce014a1e
 2015-11-10 09:55:29 -08:00
Haohui Mai
94a1833638 HDFS-2261. AOP unit tests are not getting compiled or run. Contributed by Haohui Mai. 2015-11-09 15:25:19 -08:00
Yongjun Zhang
2741a2109b HDFS-9249. NPE is thrown if an IOException is thrown in NameNode constructor. (Wei-Chiu Chuang via Yongjun Zhang) 2015-11-09 14:04:03 -08:00
Xiaoyu Yao
41d3f8899d HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. Contributed by Surendra Singh Lilhore. 2015-11-09 09:57:56 -08:00
Vinayakumar B
6d2332ae37 HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula) 2015-11-08 16:24:27 +05:30
Arpit Agarwal
2801b42a7e HDFS-9379. Make NNThroughputBenchmark support more than 10 datanodes. (Contributed by Mingliang Liu) 2015-11-06 18:58:49 -08:00
Kihwal Lee
bf6aa30a15 HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu Shukla. 2015-11-06 13:30:33 -06:00
Yongjun Zhang
b64242c0d2 HDFS-9236. Missing sanity check for block size during block recovery. (Tony Wu via Yongjun Zhang) 2015-11-06 11:15:54 -08:00
Arpit Agarwal
0b18e5e8c6 HDFS-6481. DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs. (Contributed by szetszwo) 2015-11-06 10:15:23 -08:00
Lei Xu
46e78a7a16 HDFS-9282. Make data directory count and storage raw capacity related tests FsDataset-agnostic. (Tony Wu via lei) 2015-11-05 10:47:33 -08:00
Xiaoyu Yao
ea5bb48326 HDFS-9360. Storage type usage isn't updated properly after file deletion. Contributed by Ming Ma. 2015-11-05 01:30:13 -08:00
Lei Xu
5667129276 HDFS-9363. Add fetchReplica to FsDatasetTestUtils to return FsDataset-agnostic replica. (Tony Wu via lei) 2015-11-04 10:49:28 -08:00
Lei Xu
e2a5441b06 HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for filesystem entirely allocated for DFS use. (Tony Wu via lei) 2015-11-04 10:27:35 -08:00
Lei Xu
ec414600ed HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) 2015-11-04 10:22:17 -08:00