big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,108 Commits 2 Branches 2 Tags 264 MiB
414fbfab41
Commit Graph

3225 Commits

Author SHA1 Message Date
Arpit Agarwal
2b1b2faf76 HDFS-9671. DiskBalancer: SubmitPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
66f0bb646d HDFS-9647. DiskBalancer: Add getRuntimeSettings. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
96fe685b7a HDFS-9645. DiskBalancer: Add Query RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
0501d430e2 HDFS-9595. DiskBalancer: Add cancelPlan RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
7100c0da35 HDFS-9588. DiskBalancer: Add submitDiskbalancer RPC. (Anu Engineer via Arpit Agarwal) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
5724a10316 HDFS-9469. DiskBalancer: Add Planner. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
30c6ebd699 HDFS-9449. DiskBalancer: Add connectors. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
91a5c48143 HDFS-9420. Add DataModels for DiskBalancer. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Xiaoyu Yao
dca298d79e HDFS-10469. Add number of active xceivers to datanode metrics. Contributed by Hanisha Koneru. 2016-06-23 14:13:39 -07:00
Kai Zheng
e6cb07520f HDFS-10460. Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block. Contributed by Rakesh R 2016-06-24 17:39:32 +08:00
Uma Maheswara Rao G
17eae9ebb3 HDFS-10473: Allow only suitable storage policies to be set on striped files. Contributed by Uma Maheswara Rao G 2016-06-22 11:17:43 -07:00
Akira Ajisaka
d433b16ce6 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 2016-06-22 11:00:16 +09:00
Brahma Reddy Battula
f2ac132d6a HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 2016-06-21 15:42:28 +05:30
Jitendra Pandey
7adc4d1769 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 17:22:55 -07:00
Colin Patrick Mccabe
5f6b4157a4 HDFS-10328. Add per-cache-pool default replication num configuration (xupeng via cmccabe) 2016-06-20 10:42:50 -07:00
Arpit Agarwal
0319d73c3b HDFS-10437. ReconfigurationProtocol not covered by HDFSPolicyProvider. (Arpit Agarwal) 2016-06-18 16:16:54 -07:00
Akira Ajisaka
09e82acaf9 HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 2016-06-17 16:20:38 +09:00
Vinayakumar B
cc6c265171 HDFS-10256. Use GenericTestUtils.getTestDir method in tests for temporary directories (Contributed by Vinayakumar B) 2016-06-16 16:47:06 +05:30
Colin Patrick Mccabe
2449db507d HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 2016-06-15 22:35:19 -07:00
Ming Ma
b48f27e794 HDFS-9922. Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes. (Chris Trezzo via mingma) 2016-06-15 22:00:52 -07:00
Ming Ma
7d521a29ee HDFS-9016. Display upgrade domain information in fsck. (mingma) 2016-06-14 20:05:50 -07:00
Andrew Wang
709a814fe0 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 2016-06-13 11:50:19 -07:00
Ravi Prakash
ae047655f4 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 2016-06-08 13:44:22 -07:00
Andrew Wang
1ee9ea0026 Remove redundant TestMiniDFSCluster.testDualClusters. Contributed by Jiayi Zhou. 2016-06-08 12:58:56 -07:00
Jing Zhao
be34e85e68 HDFS-10468. HDFS read ends up ignoring an interrupt. Contributed by Jing Zhao 2016-06-07 10:48:21 -07:00
Tsz-Wo Nicholas Sze
35f255b03b Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 5360da8bd9.
 2016-06-06 16:31:43 +08:00
Tsz-Wo Nicholas Sze
7e7b1ae037 Revert "Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou."" 
This reverts commit 8cf47d8589.
 2016-06-06 16:31:38 +08:00
Tsz-Wo Nicholas Sze
db41e6d285 Revert "Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou"" 
This reverts commit 5ee5912ebd.
 2016-06-06 16:31:34 +08:00
Tsz-Wo Nicholas Sze
b3d81f38da Revert "Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit b82c74b910.
 2016-06-06 16:31:30 +08:00
Tsz-Wo Nicholas Sze
cba9a01889 Revert "Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit f23d5dfc60.
 2016-06-06 16:29:38 +08:00
Tsz-Wo Nicholas Sze
aa20fa150d Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 4d36b221a2.
 2016-06-06 16:28:47 +08:00
Tsz-Wo Nicholas Sze
eded3d109e Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit 106234d873.
 2016-06-06 16:28:21 +08:00
Andrew Wang
106234d873 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit fc94810d3f.
 2016-06-03 18:09:55 -07:00
Andrew Wang
4d36b221a2 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit 1b9f18623a.
 2016-06-03 18:09:18 -07:00
Andrew Wang
f23d5dfc60 Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 7251bb922b.
 2016-06-03 18:09:17 -07:00
Andrew Wang
b82c74b910 Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 02d4e478a3.
 2016-06-03 18:09:14 -07:00
Andrew Wang
5ee5912ebd Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou" 
This reverts commit f4b9bcd87c.
 2016-06-03 18:09:13 -07:00
Andrew Wang
8cf47d8589 Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou." 
This reverts commit 21890c4239.
 2016-06-03 18:09:12 -07:00
Andrew Wang
5360da8bd9 Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit 83f2f78c11.
 2016-06-03 18:09:09 -07:00
Arpit Agarwal
97e2449477 HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (Contributed by Akira Ajisaka) 2016-06-02 13:14:45 -07:00
Akira Ajisaka
1df6f5735c HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 2016-06-03 04:10:32 +09:00
Masatake Iwasaki
aadb77e412 HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 2016-06-02 20:55:04 +09:00
Akira Ajisaka
69555fca06 HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail. Contributed by Masatake Iwasaki. 2016-06-02 18:52:47 +09:00
Kai Zheng
d749cf65e1 HDFS-9833. Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data. Contributed by Rakesh R. 2016-06-02 12:56:21 +08:00
Colin Patrick Mccabe
8ceb06e239 HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:54:34 -07:00
Colin Patrick Mccabe
c7921c9bdd HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:37:52 -07:00
Colin Patrick Mccabe
29d6cadc52 HDFS-10415. TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called (Mingliang Liu via cmccabe) 2016-05-31 16:54:52 -07:00
Tsz-Wo Nicholas Sze
83f2f78c11 HADOOP-13226 Support async call retry and failover. 2016-05-31 16:30:11 -07:00
Kai Zheng
93d8a7f2a2 Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng" 
This reverts commit a6c79f92d5.
 2016-06-01 08:41:15 +08:00
Chris Nauroth
21890c4239 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 2016-05-27 14:20:18 -07:00
Yongjun Zhang
5ea6fd85c7 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 2016-05-27 10:02:02 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Karthik Kambatla
55c3e2de3d HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:24:31 -07:00
Tsz-Wo Nicholas Sze
f4b9bcd87c HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:15:17 -07:00
Kai Zheng
77202fa103 HADOOP-13010. Refactor raw erasure coders. Contributed by Kai Zheng 2016-05-27 13:23:34 +08:00
Kai Zheng
f69f5ab3b6 HDFS-10434. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-05-27 04:42:53 +08:00
Kai Zheng
a6c79f92d5 HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng 2016-05-27 04:29:42 +08:00
Tsz-Wo Nicholas Sze
02d4e478a3 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-24 14:51:27 -07:00
Tsz-Wo Nicholas Sze
15ed080e36 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 12:49:48 -07:00
Karthik Kambatla
57c31a3fef HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha) 2016-05-24 10:37:00 -07:00
Allen Wittenauer
4b0f55b6ea HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw) 2016-05-23 15:52:01 -07:00
Chris Nauroth
500e946729 HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 2016-05-20 13:01:43 -07:00
Chris Nauroth
bcde1562d2 HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 2016-05-20 12:47:41 -07:00
Kihwal Lee
204de430bf HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-19 15:52:32 -05:00
Andrew Wang
010e6ac328 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor. 2016-05-18 14:13:41 -07:00
Lei Xu
cf552aa87b HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei) 2016-05-18 09:38:37 -07:00
Arpit Agarwal
f4d8fde822 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:20:57 -07:00
Xiaoyu Yao
dd99f5fc79 HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 16:12:00 -07:00
Yongjun Zhang
e24fe2641b HDFS-9732, Improve DelegationTokenIdentifier.toString() for better logging. Contributed by Yongjun Zhang. 2016-05-17 14:03:08 -07:00
Kihwal Lee
4a5819dae2 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 2016-05-17 08:53:31 -05:00
Akira Ajisaka
9fe5828f05 HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 2016-05-17 15:53:09 +09:00
Sangjin Lee
61f46be071 HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee) 2016-05-16 18:49:47 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Kai Zheng
ad9441122f HDFS-8449. Add tasks count metrics to datanode for ECWorker. Contributed by Bo Li. 2016-05-15 23:39:09 -07:00
Andrew Wang
45788204ae HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 2016-05-15 22:15:05 -07:00
Lei Xu
a2774debf7 HDFS-9389. Add maintenance states to AdminStates. (Ming Ma via lei) 2016-05-12 15:42:54 -07:00
Tsz-Wo Nicholas Sze
7251bb922b HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-11 17:53:40 -07:00
Colin Patrick Mccabe
687233f20d HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 2016-05-11 13:45:39 -07:00
Zhe Zhang
307db61aac HDFS-10172. hdfs erasurecode command should remove the redundant -usage option. Contributed by Yuanbo Liu. 2016-05-10 15:20:17 -07:00
Kihwal Lee
b9e5a32fa1 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 2016-05-09 15:50:09 -05:00
Akira Ajisaka
411fb4b2b7 HDFS-10337. OfflineEditsViewer stats option should print 0 instead of null for the count of operations. Contributed by Yiqun Lin. 2016-05-08 15:49:13 -07:00
Masatake Iwasaki
2835f146b5 HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 2016-05-06 15:36:47 +09:00
Xiaoyu Yao
dacd1f50fe HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 2016-05-05 18:08:47 -07:00
Ravi Prakash
d8faf47f32 HADOOP-13051. Test for special characters in path being respected during globPaths. Contributed by Harsh J. 2016-05-05 14:18:39 -07:00
Ming Ma
1268cf5fbe HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 2016-05-04 17:02:26 -07:00
Arpit Agarwal
6d77d6eab7 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:52:43 -07:00
Xiaoyu Yao
45a753ccf7 HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 2016-05-02 19:30:47 -07:00
Tsz-Wo Nicholas Sze
1b9f18623a HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:16:30 -07:00
Kihwal Lee
af9b000535 HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 2016-04-29 09:58:44 -05:00
Kihwal Lee
6243eabb48 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla 2016-04-28 12:44:53 -05:00
Lei Xu
0a152103f1 HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 2016-04-27 14:22:51 -07:00
Kihwal Lee
919a1d824a HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 2016-04-27 08:19:48 -05:00
Tsz-Wo Nicholas Sze
fc94810d3f HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-04-26 17:10:13 -07:00
Zhe Zhang
5865fe2bf0 HDFS-9869. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]. Contributed by Rakesh R. 2016-04-25 22:01:54 -07:00
Masatake Iwasaki
8eadd7145e HDFS-10325. TestDFSShell#testMoveWithTargetPortEmpty fails. Contributed by Takanobu Asanuma. 2016-04-26 07:11:13 +09:00
Tsz-Wo Nicholas Sze
f308561f1d HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-04-25 12:01:48 -07:00
Kihwal Lee
10f0f7851a HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 2016-04-25 11:38:14 -05:00
Masatake Iwasaki
6fcde2e38d HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 2016-04-23 23:37:56 +09:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Xiaoyu Yao
b4be288c5d HDFS-10207. Support enable Hadoop IPC backoff without namenode restart. Contributed by Xiaobing Zhou. 2016-04-21 10:18:48 -07:00