big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13,730 Commits 2 Branches 2 Tags 264 MiB
7adc4d1769
Commit Graph

5570 Commits

Author SHA1 Message Date
Kai Zheng
93d8a7f2a2 Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng" 
This reverts commit a6c79f92d5.
 2016-06-01 08:41:15 +08:00
Chris Nauroth
21890c4239 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 2016-05-27 14:20:18 -07:00
Yongjun Zhang
5ea6fd85c7 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 2016-05-27 10:02:02 -07:00
Kai Zheng
916140604f HADOOP-12911. Upgrade Hadoop MiniKDC with Kerby. Contributed by Jiajia Li 2016-05-28 14:23:39 +08:00
Zhe Zhang
8c84a2a93c HDFS-10236. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-3]. Contributed by Rakesh R. 2016-05-26 16:50:55 -07:00
Karthik Kambatla
55c3e2de3d HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:24:31 -07:00
Tsz-Wo Nicholas Sze
f4b9bcd87c HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:15:17 -07:00
Kai Zheng
77202fa103 HADOOP-13010. Refactor raw erasure coders. Contributed by Kai Zheng 2016-05-27 13:23:34 +08:00
Kai Zheng
f69f5ab3b6 HDFS-10434. Fix intermittent test failure of TestDataNodeErasureCodingMetrics. Contributed by Rakesh R 2016-05-27 04:42:53 +08:00
Kai Zheng
a6c79f92d5 HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng 2016-05-27 04:29:42 +08:00
Vinayakumar B
2dab64da2f HDFS-10217. show 'blockScheduled' tooltip in datanodes table. 2016-05-25 12:34:52 +05:30
Tsz-Wo Nicholas Sze
02d4e478a3 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-24 14:51:27 -07:00
Tsz-Wo Nicholas Sze
15ed080e36 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 12:49:48 -07:00
Karthik Kambatla
57c31a3fef HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha) 2016-05-24 10:37:00 -07:00
Allen Wittenauer
4b0f55b6ea HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw) 2016-05-23 15:52:01 -07:00
Masatake Iwasaki
12eb5a7b11 HDFS-10439. Update setOwner doc in HdfsPermissionsGuide. Contributed by John Zhuge. 2016-05-22 16:53:30 +09:00
Chris Nauroth
500e946729 HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 2016-05-20 13:01:43 -07:00
Chris Nauroth
bcde1562d2 HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 2016-05-20 12:47:41 -07:00
Tsz-Wo Nicholas Sze
42c22f7e3d HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-19 15:34:04 -07:00
Kihwal Lee
204de430bf HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-19 15:52:32 -05:00
Andrew Wang
010e6ac328 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor. 2016-05-18 14:13:41 -07:00
Lei Xu
cf552aa87b HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei) 2016-05-18 09:38:37 -07:00
Arpit Agarwal
f4d8fde822 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:20:57 -07:00
Xiaoyu Yao
dd99f5fc79 HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 16:12:00 -07:00
Yongjun Zhang
16c07cc68a HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang) 2016-05-17 15:57:33 -07:00
Junping Du
f6ef876fe1 MAPREDUCE-6657. Job history server can fail on startup when NameNode is in start phase. Contributed by Haibo Chen. 2016-05-17 14:41:47 -07:00
Yongjun Zhang
e24fe2641b HDFS-9732, Improve DelegationTokenIdentifier.toString() for better logging. Contributed by Yongjun Zhang. 2016-05-17 14:03:08 -07:00
Kihwal Lee
1356cbe994 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit 0942954e8a.
 2016-05-17 15:20:03 -05:00
Kihwal Lee
0942954e8a HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-17 15:06:25 -05:00
Colin Patrick Mccabe
7cd5ae62f6 HDFS-10404. Fix formatting of CacheAdmin command usage help text (Yiqun Lin via cmccabe) 2016-05-17 11:10:48 -07:00
Kihwal Lee
4a5819dae2 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 2016-05-17 08:53:31 -05:00
Akira Ajisaka
9fe5828f05 HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 2016-05-17 15:53:09 +09:00
Sangjin Lee
61f46be071 HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee) 2016-05-16 18:49:47 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Lei Xu
6a6e74acf5 HDFS-10410. RedundantEditLogInputStream.LOG is set to wrong class. (John Zhuge via lei) 2016-05-16 17:05:46 -07:00
Kai Zheng
ad9441122f HDFS-8449. Add tasks count metrics to datanode for ECWorker. Contributed by Bo Li. 2016-05-15 23:39:09 -07:00
Andrew Wang
45788204ae HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 2016-05-15 22:15:05 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Lei Xu
a2774debf7 HDFS-9389. Add maintenance states to AdminStates. (Ming Ma via lei) 2016-05-12 15:42:54 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Tsz-Wo Nicholas Sze
7251bb922b HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-11 17:53:40 -07:00
Colin Patrick Mccabe
687233f20d HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 2016-05-11 13:45:39 -07:00
Zhe Zhang
307db61aac HDFS-10172. hdfs erasurecode command should remove the redundant -usage option. Contributed by Yuanbo Liu. 2016-05-10 15:20:17 -07:00
Colin Patrick Mccabe
f0252ad252 HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe) 2016-05-10 11:15:05 -07:00
Kihwal Lee
b9e5a32fa1 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 2016-05-09 15:50:09 -05:00
Akira Ajisaka
411fb4b2b7 HDFS-10337. OfflineEditsViewer stats option should print 0 instead of null for the count of operations. Contributed by Yiqun Lin. 2016-05-08 15:49:13 -07:00
Masatake Iwasaki
6957e45699 HDFS-10374. typo in HDFSHighAvailabilityWithQJM doc. Contributed by Takashi Ohnishi. 2016-05-07 11:26:42 +09:00
Masatake Iwasaki
2835f146b5 HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 2016-05-06 15:36:47 +09:00
Xiaoyu Yao
dacd1f50fe HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 2016-05-05 18:08:47 -07:00
Ravi Prakash
d8faf47f32 HADOOP-13051. Test for special characters in path being respected during globPaths. Contributed by Harsh J. 2016-05-05 14:18:39 -07:00
Allen Wittenauer
35cf503149 HADOOP-13077. Handle special characters in passwords in httpfs.sh (Xiao Chen via aw) 2016-05-05 11:33:06 -07:00
Ming Ma
1268cf5fbe HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 2016-05-04 17:02:26 -07:00
Arpit Agarwal
6d77d6eab7 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:52:43 -07:00
Xiaoyu Yao
45a753ccf7 HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 2016-05-02 19:30:47 -07:00
Andrew Wang
3ff0510ffd Fix hadoop-hdfs-native-client compilation on Windows. Contributed by Brahma Reddy Battula. 2016-05-02 18:15:51 -07:00
Tsz-Wo Nicholas Sze
1b9f18623a HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:16:30 -07:00
Ravi Prakash
2753185a01 HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano 2016-04-29 22:32:27 -07:00
Kihwal Lee
7da540d03e HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 2016-04-29 15:50:43 -05:00
Tsz-Wo Nicholas Sze
4da6f69ca1 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:22:33 -07:00
Kihwal Lee
af9b000535 HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 2016-04-29 09:58:44 -05:00
Kihwal Lee
6243eabb48 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla 2016-04-28 12:44:53 -05:00
Lei Xu
8bd0bca0b1 HDFS-3702. Fix missing imports from HDFS-3702 trunk patch. 2016-04-27 15:37:41 -07:00
Lei Xu
0a152103f1 HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 2016-04-27 14:22:51 -07:00
Allen Wittenauer
7b1c37a13a HADOOP-12892. fix/rewrite create-release (aw) 2016-04-27 08:38:22 -07:00
Kihwal Lee
919a1d824a HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 2016-04-27 08:19:48 -05:00
Tsz-Wo Nicholas Sze
fc94810d3f HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-04-26 17:10:13 -07:00
Andrew Wang
6be22ddbf1 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-26 13:04:18 -07:00
Kihwal Lee
c6a2430b47 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 2016-04-26 11:45:29 -05:00
Zhe Zhang
5865fe2bf0 HDFS-9869. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]. Contributed by Rakesh R. 2016-04-25 22:01:54 -07:00
Masatake Iwasaki
8eadd7145e HDFS-10325. TestDFSShell#testMoveWithTargetPortEmpty fails. Contributed by Takanobu Asanuma. 2016-04-26 07:11:13 +09:00
Tsz-Wo Nicholas Sze
f308561f1d HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-04-25 12:01:48 -07:00
Kihwal Lee
10f0f7851a HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 2016-04-25 11:38:14 -05:00
Masatake Iwasaki
6fcde2e38d HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 2016-04-23 23:37:56 +09:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Ravi Prakash
d6402faded Revert "HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano" 
This reverts commit 4838b735f0.
 2016-04-22 10:17:20 -07:00
Kihwal Lee
5d4255a801 HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 2016-04-22 11:08:44 -05:00
Jing Zhao
4a043b3fec HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou. 2016-04-21 20:17:27 -07:00
Tsz-Wo Nicholas Sze
14ab7a81e2 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:41:50 -07:00
Tsz-Wo Nicholas Sze
bbce1d525e HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:20:12 -07:00
Ravi Prakash
4838b735f0 HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano 2016-04-21 11:32:39 -07:00
Xiaoyu Yao
b4be288c5d HDFS-10207. Support enable Hadoop IPC backoff without namenode restart. Contributed by Xiaobing Zhou. 2016-04-21 10:18:48 -07:00
Xiaoyu Yao
5bd7b592e5 HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao. 2016-04-20 23:10:52 -07:00
Chris Nauroth
63ac2db59a HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 2016-04-20 13:39:44 -07:00
Colin Patrick Mccabe
ad36fa6f42 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 2016-04-20 11:33:11 -07:00
Arpit Agarwal
af9bdbe447 HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:26:30 -07:00
Walter Su
90d1af1416 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 2016-04-19 18:54:11 +08:00
Walter Su
4b4df2cd48 HDFS-10284. o.a.h.hdfs.server.blockmanagement.TestBlockManagerSafeMode.testCheckSafeMode fails intermittently. Contributed by Mingliang Liu. 2016-04-19 18:07:46 +08:00
Steve Loughran
438efc3634 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 
(cherry picked from commit dd64769699ea4c85abdf9c590b9db680837e8b27)
 2016-04-19 17:48:20 +08:00
Kai Zheng
b5d4c7dc76 HADOOP-12924. Configure raw erasure coders for supported codecs. Contributed by Rui Li. 2016-04-20 15:47:01 +08:00
Jing Zhao
be0bce1b71 HDFS-10306. SafeModeMonitor should not leave safe mode if name system is starting active service. Contributed by Mingliang Liu. 2016-04-18 18:05:49 -07:00
Colin Patrick Mccabe
cb3ca460ef HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 2016-04-18 13:47:56 -07:00
Kihwal Lee
d8b729e16f HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 2016-04-18 07:58:55 -05:00
Walter Su
ab903029a9 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 2016-04-18 20:29:29 +08:00
Walter Su
67523ffcf4 HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 2016-04-18 09:28:02 +08:00
Jing Zhao
55e19b7f0c HDFS-10293. StripedFileTestUtil#readAll flaky. Contributed by Mingliang Liu. 2016-04-15 10:53:40 -07:00
Jing Zhao
89a838769f HDFS-10283. o.a.h.hdfs.server.namenode.TestFSImageWithSnapshot#testSaveLoadImageWithAppending fails intermittently. Contributed by Mingliang Liu. 2016-04-15 10:49:21 -07:00
Kihwal Lee
b9c9d03591 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 2016-04-14 15:24:39 -05:00
Kihwal Lee
2c155afe27 HDFS-10292. Add block id when client got Unable to close file exception. Contributed by Brahma Reddy Battula. 2016-04-14 14:25:11 -05:00
Xiaoyu Yao
809226752d HDFS-10286. Fix TestDFSAdmin#testNameNodeGetReconfigurableProperties. Contributed by Xiaobing Zhou. 2016-04-14 10:57:02 -07:00