big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,563 Commits 2 Branches 2 Tags 264 MiB
5cb1e0118b
Commit Graph

4446 Commits

Author SHA1 Message Date
Haohui Mai
b21dffb1fe HDFS-9451. Clean up depreated umasks and related unit tests. Contributed by Wei-Chiu Chuang. 2015-11-25 12:47:24 -08:00
Yongjun Zhang
8176ea7dc6 HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. (John Zhuge via Yongjun Zhang) 2015-11-25 07:40:16 -08:00
Tsz-Wo Nicholas Sze
78ec38b2ed HDFS-8807. dfs.datanode.data.dir does not handle spaces between storageType and URI correctly. Contributed by Anu Engineer 2015-11-24 16:01:55 -08:00
Haohui Mai
977e0b3c4c HDFS-8335. FSNamesystem should construct FSPermissionChecker only if permission is enabled. Contributed by Gabor Liptak. 2015-11-24 13:14:49 -08:00
Xiaoyu Yao
fe5624b85d HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou. 2015-11-24 12:47:57 -08:00
Haohui Mai
db4cab21f4 HDFS-5165. Remove the TotalFiles metrics. Contributed by Akira AJISAKA. 2015-11-24 11:41:57 -08:00
Ming Ma
0e54b164a8 HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess replicas. (Xiao Chen via mingma) 2015-11-24 10:30:24 -08:00
cnauroth
1777608fa0 HDFS-6101. TestReplaceDatanodeOnFailure fails occasionally. Contributed by Wei-Chiu Chuang. 2015-11-24 09:39:21 -08:00
Uma Mahesh
411e2b2e7c HDFS-9433. DFS getEZForPath API on a non-existent file should throw FileNotFoundException (Rakesh R via umamahesh) 2015-11-23 17:54:31 -08:00
Walter Su
201f14e271 HDFS-9423. Fix intermittent failure of TestEditLogTailer. Contributed by Masatake Iwasaki. 2015-11-23 18:49:40 +08:00
Walter Su
001ecf962c HDFS-9435. TestBlockRecovery#testRBWReplicas is failing intermittently. Contributed by Rakesh R. 2015-11-23 15:49:58 +08:00
Walter Su
5aba093361 HDFS-9428. Fix intermittent failure of TestDNFencing.testQueueingWithAppend. Contributed by Masatake Iwasaki. 2015-11-23 15:37:45 +08:00
Haohui Mai
8228697857 HDFS-9024. Deprecate the TotalFiles metric. Contributed by Akira Ahisaka. 2015-11-22 23:02:50 -08:00
Haohui Mai
04c14b5dc4 HDFS-9356. Decommissioning node does not have Last Contact value in the UI. Contributed by Surendra Singh Lilhore. 2015-11-22 22:58:58 -08:00
Haohui Mai
a4bd54f9d7 HDFS-7897. Shutdown metrics when stopping JournalNode. Contributed by zhouyingchao. 2015-11-22 19:12:23 -08:00
Haohui Mai
bfbcfe7364 HDFS-6885. Fix wrong use of BytesWritable in FSEditLogOp#RenameOp. Contributed by Yi Liu. 2015-11-22 17:16:51 -08:00
Haohui Mai
2326171ea8 HDFS-3302. Review and improve HDFS trash documentation. Contributed by Harsh J. 2015-11-22 16:30:39 -08:00
Haohui Mai
38146a6cdb HDFS-7796. Include X-editable for slick contenteditable fields in the web UI. Contributed by Ravi Prakash. 2015-11-22 16:22:06 -08:00
Haohui Mai
0c7340f377 HDFS-8914. Document HA support in the HDFS HdfsDesign.md. Contributed by Lars Francke. 2015-11-22 16:10:27 -08:00
Haohui Mai
176ff5ce90 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:54:41 -08:00
Uma Mahesh
5b714a2819 HDFS-9348. Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent file should be handled properly. (Rakesh R via umamahesh) 2015-11-21 22:38:36 -08:00
Tsz-Wo Nicholas Sze
8e03e855b6 HDFS-9434. Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags. 2015-11-19 14:54:01 -08:00
Yongjun Zhang
31dc5c650e HDFS-9439. Include status of closeAck into exception message in DataNode#run. (Xiao Chen via Yongjun Zhang) 2015-11-19 14:20:33 -08:00
Ming Ma
1c4951a7a0 HDFS-8056. Decommissioned dead nodes should continue to be counted as dead after NN restart. (mingma) 2015-11-19 10:04:01 -08:00
Akira Ajisaka
ac1aa6c819 HDFS-9431. DistributedFileSystem#concat fails if the target path is relative. Contributed by Kazuho Fujii. 2015-11-19 20:55:01 +09:00
Colin Patrick Mccabe
dfbde3fc51 HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp. (Lei (Eddy) Xu via cmccabe) 2015-11-17 10:48:15 -08:00
Arpit Agarwal
007c6ce1af HDFS-9397. Fix typo for readChecksum() LOG.warn in BlockSender.java. (Contributed by Enrique Flores) 2015-11-16 21:02:00 -08:00
Walter Su
621cbb4f69 HDFS-9358. TestNodeCount#testNodeCount timed out. Contributed by Masatake Iwasaki. 2015-11-17 10:51:29 +08:00
Xiaoyu Yao
a4f62a2d58 HDFS-9421. NNThroughputBenchmark replication test NPE with -namenode option. Contributed by Mingliang Liu. 2015-11-16 16:42:15 -08:00
Xiaoyu Yao
2701f2d255 HDFS-9387. Fix namenodeUri parameter parsing in NNThroughputBenchmark. Contributed by Mingliang Liu. 2015-11-16 16:29:47 -08:00
Ming Ma
02653add98 HDFS-9413. getContentSummary() on standby should throw StandbyException. (Brahma Reddy Battula via mingma) 2015-11-16 09:32:40 -08:00
Walter Su
cccf88480b HDFS-9410. Some tests should always reset sysout and syserr. Contributed by Xiao Chen. 2015-11-13 14:26:43 +08:00
Zhe Zhang
7b00c8e20e HDFS-8968. Erasure coding: a comprehensive I/O throughput benchmark tool. Contributed by Rui Li. 
Change-Id: I01ff5b04727fd79e6373582d4815c5e7b2096c67
 2015-11-12 11:30:39 -08:00
Kihwal Lee
a2a104f3d3 HDFS-9396. Total files and directories on jmx and web UI on standby is uninitialized. Contributed by Kihwal Lee. 2015-11-11 15:24:43 -06:00
Zhe Zhang
73b94d7899 HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. Contributed by Xiao Chen. 
Change-Id: I9e42f724f27924cf73891425a832de37ce014a1e
 2015-11-10 09:55:29 -08:00
Walter Su
2fda45b9dc HDFS-9401. Fix findbugs warnings in BlockRecoveryWorker. Contributed by Brahma Reddy Battula. 2015-11-10 14:48:08 +08:00
Haohui Mai
94a1833638 HDFS-2261. AOP unit tests are not getting compiled or run. Contributed by Haohui Mai. 2015-11-09 15:25:19 -08:00
Yongjun Zhang
2741a2109b HDFS-9249. NPE is thrown if an IOException is thrown in NameNode constructor. (Wei-Chiu Chuang via Yongjun Zhang) 2015-11-09 14:04:03 -08:00
Xiaoyu Yao
41d3f8899d HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. Contributed by Surendra Singh Lilhore. 2015-11-09 09:57:56 -08:00
Vinayakumar B
6d2332ae37 HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula) 2015-11-08 16:24:27 +05:30
Allen Wittenauer
b76b0ce51e HADOOP-12366. expose calculated paths (aw) 2015-11-07 08:32:56 -08:00
Arpit Agarwal
2801b42a7e HDFS-9379. Make NNThroughputBenchmark support more than 10 datanodes. (Contributed by Mingliang Liu) 2015-11-06 18:58:49 -08:00
Kihwal Lee
bf6aa30a15 HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu Shukla. 2015-11-06 13:30:33 -06:00
Yongjun Zhang
b64242c0d2 HDFS-9236. Missing sanity check for block size during block recovery. (Tony Wu via Yongjun Zhang) 2015-11-06 11:15:54 -08:00
Arpit Agarwal
0b18e5e8c6 HDFS-6481. DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs. (Contributed by szetszwo) 2015-11-06 10:15:23 -08:00
Yongjun Zhang
286cc6483f HDFS-9377. Fix findbugs warnings in FSDirSnapshotOp. (Mingliang Liu via Yongjun Zhang) 2015-11-05 14:36:12 -08:00
Lei Xu
46e78a7a16 HDFS-9282. Make data directory count and storage raw capacity related tests FsDataset-agnostic. (Tony Wu via lei) 2015-11-05 10:47:33 -08:00
Haohui Mai
efc73d7896 HDFS-9372. Remove dead code in DataStorage.recoverTransitionRead. Contributed by Duo Zhang. 2015-11-05 10:00:01 -08:00
Kihwal Lee
ff47f35dee HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee. 2015-11-05 09:25:20 -06:00
Xiaoyu Yao
ea5bb48326 HDFS-9360. Storage type usage isn't updated properly after file deletion. Contributed by Ming Ma. 2015-11-05 01:30:13 -08:00
Lei Xu
5667129276 HDFS-9363. Add fetchReplica to FsDatasetTestUtils to return FsDataset-agnostic replica. (Tony Wu via lei) 2015-11-04 10:49:28 -08:00
Lei Xu
e2a5441b06 HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for filesystem entirely allocated for DFS use. (Tony Wu via lei) 2015-11-04 10:27:35 -08:00
Lei Xu
ec414600ed HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) 2015-11-04 10:22:17 -08:00
Haohui Mai
88beb46cf6 Revert "HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou." 
This reverts commit 84cbd72afd.
 2015-11-04 10:21:13 -08:00
Haohui Mai
0eed886a16 HDFS-9357. NN UI renders icons of decommissioned DN incorrectly. Contributed by Surendra Singh Lilhore. 2015-11-04 09:16:43 -08:00
Varun Vasudev
73b9c7b82b HADOOP-10787. Rename/remove non-HADOOP_*, etc from the shell scripts. Contributed by Allen Wittenauer. 2015-11-04 15:56:17 +05:30
Yongjun Zhang
194251c852 HDFS-9351. checkNNStartup() need to be called when fsck calls FSNamesystem.getSnapshottableDirs(). (Xiao Chen via Yongjun Zhang) 2015-11-03 17:16:17 -08:00
Zhe Zhang
dac0463a4e HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. Contributed by Chang Li. 
Change-Id: If5ce1b2d212bb0726bce52ad12a3de401bcec02d
 2015-11-03 13:34:24 -08:00
cnauroth
7e2829662b HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line endings, fails on Windows. Contributed by Chris Nauroth. 2015-11-03 11:54:57 -08:00
cnauroth
095ac83402 HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows. Contributed by Xiaoyu Yao. 2015-11-03 11:21:08 -08:00
Ming Ma
d565480da2 HDFS-9313. Possible NullPointerException in BlockManager if no excess replica can be chosen. (mingma) 2015-11-02 19:36:37 -08:00
Lei Xu
8e05dbf2bd HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony Wu via lei) 2015-11-02 18:05:43 -08:00
yliu
5ba2b98d0f HDFS-9275. Wait previous ErasureCodingWork to finish before schedule another one. (Walter Su via yliu) 2015-11-03 09:14:32 +08:00
Lei Xu
7632409482 HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei) 2015-11-02 17:09:39 -08:00
Zhe Zhang
78d6890865 HDFS-9339. Extend full test of KMS ACLs. Contributed by Daniel Templeton. 
Change-Id: I61eeee8fa5e85250eabc1eef3d8c11f32700d6fb
 2015-11-02 13:51:45 -08:00
Zhe Zhang
259bea3b48 HDFS-9329. TestBootstrapStandby#testRateThrottling is flaky because fsimage size is smaller than IO buffer size. Contributed by Zhe Zhang. 
Change-Id: I09896c46e9ee0718b67c64fac5acfb3f7decf0b9
 2015-11-02 10:03:39 -08:00
Zhe Zhang
90e1405516 HDFS-8777. Erasure Coding: add tests for taking snapshots on EC files. Contributed by Rakesh R. 
Change-Id: Ia3ae582405e741ca8e90d9255ab9b95d085e5fa8
 2015-11-02 07:48:30 -08:00
Arpit Agarwal
3cde6931cb HDFS-9343. Empty caller context considered invalid. (Contributed by Mingliang Liu) 2015-11-01 15:35:02 -08:00
yliu
7fd6416759 Revert "HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee." 
This reverts commit 43539b5ff4.
 2015-10-31 16:20:48 +08:00
Kihwal Lee
43539b5ff4 HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee. 2015-10-30 09:29:13 -05:00
Tsz-Wo Nicholas Sze
f072eb5a20 HDFS-9323. Randomize the DFSStripedOutputStreamWithFailure tests. 2015-10-30 15:41:03 +08:00
Andrew Wang
888c6245e2 HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while saving namespace. 2015-10-29 11:30:47 -07:00
Zhe Zhang
8def51a708 HDFS-9229. Expose size of NameNode directory as a metric. Contributed by Surendra Singh Lilhore. 
Change-Id: I985627a5d1400249d72d084283ef366d5ac6e07b
 2015-10-29 11:23:08 -07:00
Walter Su
5eca6dece6 HADOOP-12327. Initialize output buffers with ZERO bytes in erasure coder. Contributed by Kai Zheng. 2015-10-29 15:04:34 +08:00
Vinayakumar B
588baab160 HDFS-9044. Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group (Contributed by J.Andreina) 2015-10-29 11:44:01 +05:30
Vinayakumar B
7d2d16f4ee HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path (Contributed by J.Andreina) 2015-10-29 11:28:17 +05:30
Haohui Mai
65f53f246a HDFS-9168. Move client side unit test to hadoop-hdfs-client. Contributed by Haohui Mai. 2015-10-28 15:54:37 -07:00
Zhe Zhang
73bc65e02b HDFS-9295. Add a thorough test of the full KMS code path. Contributed by Daniel Templeton. 
Change-Id: Ife1abb2f8e9a30102c4f705c47646d9ac8328786
 2015-10-28 14:36:20 -07:00
Kihwal Lee
19a77f5466 HDFS-9279. Decomissioned capacity should not be considered for configured/used capacity. Contributed by Kihu Shukla . 2015-10-28 11:58:51 -05:00
Zhe Zhang
e287e7d14b HDFS-9255. Consolidate block recovery related implementation into a single class. Contributed by Walter Su. 
Change-Id: I7a1c03f50123d79ac0a78c981d9721617e3229d1
 2015-10-28 07:34:06 -07:00
yliu
c201cf951d HADOOP-12040. Adjust inputs order for the decode API in raw erasure coder. (Kai Zheng via yliu) 2015-10-28 16:18:23 +08:00
yliu
6ff6663f64 HDFS-9302. WebHDFS throws NullPointerException if newLength is not provided. (Jagadesh Kiran N via yliu) 2015-10-28 15:54:04 +08:00
Yongjun Zhang
97913f430c HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot. (Xiao Chen via Yongjun Zhang) 2015-10-27 23:31:50 -07:00
cnauroth
bf8e452982 HDFS-9311. Support optional offload of NameNode HA service health checks to a separate RPC server. Contributed by Chris Nauroth. 2015-10-27 23:07:14 -07:00
Allen Wittenauer
1396867b52 HADOOP-12494. fetchdt stores the token based on token kind instead of token service (HeeSoo Kim via aw) 2015-10-27 12:02:43 -07:00
Colin Patrick Mccabe
fe93577faf HDFS-9299. Give ReplicationMonitor a readable thread name (Staffan Friberg via Colin P. McCabe) 2015-10-27 11:37:26 -07:00
Ming Ma
aa09880ab8 HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario. (Mingliang Liu via mingma) 2015-10-27 09:28:40 -07:00
Lei Xu
399ad00915 HDFS-9292. Make TestFileConcorruption independent to underlying FsDataset Implementation. (lei) 2015-10-26 16:09:22 -07:00
Arpit Agarwal
d8736eb9ca HDFS-9305. Delayed heartbeat processing causes storm of subsequent heartbeats. (Contributed by Arpit Agarwal) 2015-10-26 15:54:14 -07:00
Andrew Wang
e8aefdf08b HDFS-8945. Update the description about replica placement in HDFS Architecture documentation. Contributed by Masatake Iwasaki. 2015-10-26 15:47:59 -07:00
Lei Xu
37bf6141f1 HDFS-9291. Fix TestInterDatanodeProtocol to be FsDataset-agnostic. (lei) 2015-10-26 15:18:24 -07:00
Andrew Wang
677a936bf7 HDFS-9284. fsck command should not print exception trace when file not found. Contributed by Jagadesh Kiran N. 2015-10-26 15:15:03 -07:00
Yongjun Zhang
5e718de522 HDFS-7284. Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. (Wei-Chiu Chuang via Yongjun Zhang) 2015-10-26 13:25:31 -07:00
Haohui Mai
67e3d75aed HDFS-9304. Add HdfsClientConfigKeys class to TestHdfsConfigFields#configurationClasses. Contributed by Mingliang Liu. 2015-10-26 12:02:52 -07:00
Arpit Agarwal
86c92227fc HDFS-4015. Safemode should count and report orphaned blocks. (Contributed by Anu Engineer) 2015-10-23 18:07:17 -07:00
Lei Xu
5679e46b7f HDFS-9297. Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile(). (Tony Wu via lei) 2015-10-23 17:42:23 -07:00
Haohui Mai
15eb84b37e HDFS-9301. HDFS clients can't construct HdfsConfiguration instances. Contributed by Mingliang Liu. 2015-10-23 14:53:34 -07:00
Zhe Zhang
ab3c4cff4a HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby. Contributed by Zhe Zhang. 2015-10-23 14:01:49 -07:00
Lei Xu
533a2be5ac HDFS-9264. Minor cleanup of operations on FsVolumeList#volumes. (Walter Su via lei) 2015-10-23 13:52:59 -07:00
Jitendra Pandey
600ad7bf41 HDFS-9184. Logging HDFS operation's caller context into audit logs. Contributed by Mingliang Liu. 2015-10-23 12:15:01 -07:00