big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,439 Commits 2 Branches 2 Tags 264 MiB
b21dffb1fe
Commit Graph

2884 Commits

Author SHA1 Message Date
Vinayakumar B
7d2d16f4ee HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path (Contributed by J.Andreina) 2015-10-29 11:28:17 +05:30
Haohui Mai
65f53f246a HDFS-9168. Move client side unit test to hadoop-hdfs-client. Contributed by Haohui Mai. 2015-10-28 15:54:37 -07:00
Zhe Zhang
73bc65e02b HDFS-9295. Add a thorough test of the full KMS code path. Contributed by Daniel Templeton. 
Change-Id: Ife1abb2f8e9a30102c4f705c47646d9ac8328786
 2015-10-28 14:36:20 -07:00
Kihwal Lee
19a77f5466 HDFS-9279. Decomissioned capacity should not be considered for configured/used capacity. Contributed by Kihu Shukla . 2015-10-28 11:58:51 -05:00
Zhe Zhang
e287e7d14b HDFS-9255. Consolidate block recovery related implementation into a single class. Contributed by Walter Su. 
Change-Id: I7a1c03f50123d79ac0a78c981d9721617e3229d1
 2015-10-28 07:34:06 -07:00
yliu
c201cf951d HADOOP-12040. Adjust inputs order for the decode API in raw erasure coder. (Kai Zheng via yliu) 2015-10-28 16:18:23 +08:00
Yongjun Zhang
97913f430c HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks are in a snapshot. (Xiao Chen via Yongjun Zhang) 2015-10-27 23:31:50 -07:00
cnauroth
bf8e452982 HDFS-9311. Support optional offload of NameNode HA service health checks to a separate RPC server. Contributed by Chris Nauroth. 2015-10-27 23:07:14 -07:00
Ming Ma
aa09880ab8 HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario. (Mingliang Liu via mingma) 2015-10-27 09:28:40 -07:00
Lei Xu
399ad00915 HDFS-9292. Make TestFileConcorruption independent to underlying FsDataset Implementation. (lei) 2015-10-26 16:09:22 -07:00
Arpit Agarwal
d8736eb9ca HDFS-9305. Delayed heartbeat processing causes storm of subsequent heartbeats. (Contributed by Arpit Agarwal) 2015-10-26 15:54:14 -07:00
Lei Xu
37bf6141f1 HDFS-9291. Fix TestInterDatanodeProtocol to be FsDataset-agnostic. (lei) 2015-10-26 15:18:24 -07:00
Haohui Mai
67e3d75aed HDFS-9304. Add HdfsClientConfigKeys class to TestHdfsConfigFields#configurationClasses. Contributed by Mingliang Liu. 2015-10-26 12:02:52 -07:00
Arpit Agarwal
86c92227fc HDFS-4015. Safemode should count and report orphaned blocks. (Contributed by Anu Engineer) 2015-10-23 18:07:17 -07:00
Lei Xu
5679e46b7f HDFS-9297. Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile(). (Tony Wu via lei) 2015-10-23 17:42:23 -07:00
Zhe Zhang
ab3c4cff4a HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to -bootstrapStandby. Contributed by Zhe Zhang. 2015-10-23 14:01:49 -07:00
Jitendra Pandey
600ad7bf41 HDFS-9184. Logging HDFS operation's caller context into audit logs. Contributed by Mingliang Liu. 2015-10-23 12:15:01 -07:00
Andrew Wang
3dadf369d5 HDFS-7087. Ability to list /.reserved. Contributed by Xiao Chen. 2015-10-21 16:58:47 -07:00
cnauroth
1b525a9c32 HDFS-9273. ACLs on root directory may be lost after NN restart. Contributed by Xiao Chen. 2015-10-21 16:39:02 -07:00
Andrew Wang
3e50f478b9 HDFS-9225. Fix intermittent test failure of TestBlockManager.testBlocksAreNotUnderreplicatedInSingleRack. Contributed by Masatake Iwasaki. 2015-10-21 14:55:23 -07:00
Jing Zhao
d806a5bf07 HDFS-9070. Allow fsck display pending replica location information for being-written blocks. Contributed by Gao Rui. 2015-10-21 14:44:05 -07:00
Ming Ma
e27c2ae8ba HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy. (Brahma Reddy Battula via mingma) 2015-10-21 08:06:58 -07:00
Andrew Wang
6c8b6f3646 HDFS-3059. ssl-server.xml causes NullPointer. Contributed by Xiao Chen. 2015-10-20 13:44:53 -07:00
Colin Patrick Mccabe
6381ddc096 HDFS-9270. TestShortCircuitLocalRead should not leave socket after unit test (Masatake Iwasaki via Colin P. McCabe) 2015-10-20 10:51:18 -07:00
Lei Xu
71e533a153 HDFS-9251. Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in the tests code. (lei) 2015-10-20 10:08:02 -07:00
yliu
9cb5d35353 HDFS-9208. Disabling atime may fail clients like distCp. (Kihwal Lee via yliu) 2015-10-20 11:33:04 +08:00
Andrew Wang
8175c4f6b9 HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. Contributed by Xiao Chen. 2015-10-19 16:12:56 -07:00
Tsuyoshi Ozawa
4c9c3955ae HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. Contributed by Brahma Reddy Battula. 2015-10-19 15:30:21 +09:00
Kihwal Lee
c7c36cbd62 HDFS-9220. Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum. Contributed by Jing Zhao. 2015-10-15 11:24:14 -05:00
Vinayakumar B
08f5de1ef5 HDFS-9157. [OEV and OIV] : Unnecessary parsing for mandatory arguements if '-h' option is specified as the only option (Contributed by nijel) 2015-10-15 17:37:06 +05:30
Tsz-Wo Nicholas Sze
5411dc559d HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) 2015-10-15 18:07:09 +08:00
Lei Xu
c80b3a804f HDFS-9188. Make block corruption related tests FsDataset-agnostic. (lei) 2015-10-14 20:18:43 -07:00
Jing Zhao
be7a0add8b HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. Contributed by Jing Zhao. 2015-10-14 16:17:49 -07:00
Lei Xu
ba3c197878 HDFS-9238. Update TestFileCreation.testLeaseExpireHardLimit() to avoid using DataNodeTestUtils.getFile(). (Tony Wu via lei) 2015-10-14 13:10:26 -07:00
Jing Zhao
2a98724342 HDFS-1172. Blocks in newly completed files are considered under-replicated too quickly. Contributed by Masatake Iwasaki. 2015-10-13 23:00:18 -07:00
Haohui Mai
d1e1925bf6 HDFS-9145. Tracking methods that hold FSNamesytemLock for too long. Contributed by Mingliang Liu. 2015-10-13 13:24:15 -07:00
Jing Zhao
da16c9b3b4 HDFS-8438. Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy. Contributed by Walter Su. 2015-10-13 11:03:37 -07:00
Jitendra Pandey
84cbd72afd HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou. 2015-10-13 09:12:04 -07:00
Vinayakumar B
39581e3be2 HDFS-9139. Enable parallel JUnit tests for HDFS Pre-commit (Contributed by Chris Nauroth and Vinayakumar B) 2015-10-13 19:00:08 +05:30
Vinayakumar B
69b025dbba HDFS-9224. TestFileTruncate fails intermittently with BindException (Contributed by Brahma Reddy Battula) 2015-10-13 11:42:12 +05:30
Lei Xu
0f5f9846ed HDFS-9006. Provide BlockPlacementPolicy that supports upgrade domain. (Ming Ma via lei) 2015-10-12 16:24:16 -07:00
cnauroth
4f6e842ba9 HDFS-9085. Show renewer information in DelegationTokenIdentifier#toString. Contributed by Zhihai Xu. 2015-10-09 14:27:13 -07:00
Andrew Wang
c32614f410 HDFS-8941. DistributedFileSystem listCorruptFileBlocks API should resolve relative path. Contributed by Rakesh R. 2015-10-09 11:57:03 -07:00
Ming Ma
de8efc65a4 HDFS-9142. Separating Configuration object for namenode(s) in MiniDFSCluster. (Siqi Li via mingma) 2015-10-09 11:10:46 -07:00
Yongjun Zhang
c11fc8a1be HDFS-9181. Better handling of exceptions thrown during upgrade shutdown. Contributed by Wei-Chiu Chuang. 2015-10-09 09:21:29 -07:00
Jing Zhao
118a35bc2e HDFS-9204. DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated. Contributed by Mingliang Liu. 2015-10-08 14:11:02 -07:00
Yongjun Zhang
1107bd399c HDFS-8164. cTime is 0 in VERSION file for newly formatted NameNode. Contributed by Xiao Chen. 2015-10-07 23:20:10 -07:00
Jing Zhao
fde729feeb HDFS-9209. Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java. Contributed by Surendra Singh Lilhore. 2015-10-07 15:17:50 -07:00
Lei Xu
6dd47d754c HDFS-9176. Fix TestDirectoryScanner#testThrottling often fails. (Daniel Templeton via lei) 2015-10-07 14:33:43 -07:00
Kihwal Lee
99e5204ff5 HDFS-9178. Slow datanode I/O can cause a wrong node to be marked bad. Contributed by Kihwal Lee. 2015-10-07 10:17:17 -05:00
Vinayakumar B
61b3547c46 HDFS-9159. [OIV] : return value of the command is not correct if invalid value specified in '-p (processor)' option (Contributed by nijel) 2015-10-07 14:29:34 +05:30
Jing Zhao
a8b4d0ff28 HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers. Contributed by Jing Zhao. 2015-10-06 10:56:10 -07:00
Colin Patrick Mccabe
2fc2b50534 HDFS-9155. OEV should treat .XML files as XML even when the file name extension is uppercase (nijel via cmccabe) 2015-10-05 12:53:58 -07:00
Lei Xu
a68b6eb0f4 HDFS-9015. Refactor TestReplicationPolicy to test different block placement policies. (Ming Ma via lei) 2015-10-02 11:42:09 -07:00
Jing Zhao
c6cafc77e6 HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh R. 2015-10-02 11:08:17 -07:00
Zhe Zhang
6c17d31528 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java

Change-Id: I53ec1c426dc988d6c4a2c87b00caef49c4057010
 2015-09-30 08:39:13 -07:00
Aaron T. Myers
071733dc69 HDFS-9001. DFSUtil.getNsServiceRpcUris() can return too many entries in a non-HA, non-federated cluster. Contributed by Daniel Templeton. 2015-09-29 18:19:31 -07:00
Colin Patrick Mccabe
850d679acb HADOOP-12447. Clean up some htrace integration issues (cmccabe) 2015-09-29 09:25:11 -07:00
Zhe Zhang
8fd5520246 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java

Change-Id: Ic7946c4ea35bed587fe879ce58b959b25ecc0823
 2015-09-29 01:39:16 -07:00
Kihwal Lee
3abbdc929b HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 2015-09-28 17:31:51 -05:00
Jing Zhao
6419900ac2 HDFS-9040. Erasure coding: coordinate data streamers in DFSStripedOutputStream. Contributed by Jing Zhao and Walter Su. 2015-09-28 14:40:27 -07:00
Lei Xu
e5992ef4df HDFS-9147. Fix the setting of visibleLength in ExternalBlockReader. (Colin P. McCabe via Lei (Eddy) Xu) 2015-09-28 13:39:00 -07:00
Lei Xu
50741cb568 HDFS-9148. Incorrect assert message in TestWriteToReplica#testWriteToTemporary (Tony Wu via Lei (Eddy) Xu) 2015-09-28 13:27:25 -07:00
Colin Patrick Mccabe
892ade689f HDFS-9080. Update htrace version to 4.0.1 (cmccabe) 2015-09-28 07:42:12 -07:00
Haohui Mai
bf37d3d80e HDFS-8053. Move DFSIn/OutputStream and related classes to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-26 11:08:25 -07:00
Colin Patrick Mccabe
7a3c381b39 HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton via Colin P. McCabe) 2015-09-26 04:09:06 -07:00
Lei Xu
67b0e967f0 HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) 2015-09-25 16:01:41 -07:00
Lei Xu
5eb237d544 HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via Lei (Eddy) Xu) 2015-09-25 15:50:19 -07:00
Colin Patrick Mccabe
4e7c6a653f HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn Sharp via Colin P. McCabe) 
Change-Id: Ib8420310e515bb98091de86ea5c4be354878d43c
 2015-09-25 15:25:42 -07:00
Yongjun Zhang
3187dbd7a8 HDFS-9123. Copying from the root to a subdirectory should be forbidden. (Wei-Chiu Chuang via Yongjun Zhang) 2015-09-24 21:36:38 -07:00
Haohui Mai
692b1a45ce HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from HdfsServerConstants to HdfsConstants. Contributed by Mingliang Liu. 2015-09-24 20:19:16 -07:00
Haohui Mai
ead1b9e680 HDFS-9131. Move config keys used by hdfs-client to HdfsClientConfigKeys. Contributed by Mingliang Liu. 2015-09-24 00:30:01 -07:00
Haohui Mai
4893adff19 HDFS-9130. Use GenericTestUtils#setLogLevel to the logging level. Contributed by Mingliang Liu. 2015-09-23 23:59:19 -07:00
Haohui Mai
06d1c9033e HDFS-9128. TestWebHdfsFileContextMainOperations and TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on Windows. Contributed by Chris Nauroth. 2015-09-23 19:38:09 -07:00
Zhe Zhang
c09dc258a8 HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy. Contributed by Vinayakumar B. 
Change-Id: Ic56da0b426f47c63dac440aef6f5fc8554f6cf13
 2015-09-23 13:34:09 -07:00
Vinayakumar B
a2c76e5f26 HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from trunk (Contributed by Surendra Singh Lilhore) 2015-09-23 15:28:41 +05:30
Haohui Mai
7c5c099324 HDFS-8733. Keep server related definition in hdfs.proto on server side. Contributed by Mingliang Liu. 2015-09-22 20:57:05 -07:00
Haohui Mai
63d9f1596c HDFS-9039. Separate client and server side methods of o.a.h.hdfs.NameNodeProxies. Contributed by Mingliang Liu. 2015-09-22 20:52:37 -07:00
Zhe Zhang
1080c37300 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java

Change-Id: I8511c4d64b0959e79129febc179845a3892fedcc
 2015-09-22 13:27:09 -07:00
Vinayakumar B
10ab7d595e HDFS-8780. Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. (Contributed by J.Andreina) 2015-09-22 12:25:35 +05:30
Haohui Mai
06022b8fdc HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to PBHelperClient. Contributed by Mingliang Liu. 2015-09-21 18:53:13 -07:00
Zhe Zhang
a9e6681edf HDFS-9091. Erasure Coding: Provide DistributedFilesystem API to getAllErasureCodingPolicies. Contributed by Rakesh R. 
Change-Id: Id30a1ce9e2ce676d00a9220a2d3f14b8d9f00527
 2015-09-21 13:24:02 -07:00
Uma Mahesh
b762199adb HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail to process remaining tasks due to one invalid ECTask. Contributed by Uma Maheswara Rao G. 2015-09-19 21:18:42 -07:00
Lei Xu
3a9c7076e8 HDFS-9004. Add upgrade domain to DatanodeInfo. Contributed by Ming Ma (via Lei (Eddy) Xu). 
Change-Id: I887c66578eebd61acc34b94f18da6e6851c609f4
 2015-09-19 18:08:09 -07:00
Jing Zhao
3f42753102 HDFS-9063. Correctly handle snapshot path for getContentSummary. Contributed by Jing Zhao. 2015-09-18 09:26:33 -07:00
Yongjun Zhang
2ff6faf954 HDFS-5802. NameNode does not check for inode type before traversing down a path. (Xiao Chen via Yongjun Zhang) 2015-09-18 07:17:30 -07:00
Vinayakumar B
92c1af1646 HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) 2015-09-18 16:37:10 +05:30
Zhe Zhang
607bda2a21 HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when specifying policy. Contributed by Zhe Zhang. 
Change-Id: I82919e55c4a0d8646d25a53aac757098a56b3029
 2015-09-17 14:52:37 -07:00
Haohui Mai
9eee97508f HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-17 14:18:00 -07:00
Steve Loughran
08024d1b0b HDFS-9073. Fix failures in TestLazyPersistLockedMemory testReleaseOnEviction(). (J.Andreina via stevel) 2015-09-16 10:12:19 +01:00
Steve Loughran
4477643749 HDFS-9072. Fix random failures in TestJMXGet. (J.Andreina via stevel) 2015-09-16 10:08:33 +01:00
Vinayakumar B
ce69c9b54c HDFS-8953. DataNode Metrics logging (Contributed by Kanaka Kumar Avvaru) 2015-09-16 00:18:29 +05:30
Ming Ma
083b44c136 HDFS-9008. Balancer#Parameters class could use a builder pattern. (Chris Trezzo via mingma) 2015-09-15 10:16:02 -07:00
Vinayakumar B
a440567491 HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) 2015-09-15 17:30:46 +05:30
Haohui Mai
76957a485b HDFS-9010. Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT config key. Contributed by Mingliang Liu. 2015-09-14 18:22:52 -07:00
Colin Patrick Mccabe
7b5cf5352e HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi via Colin P. McCabe) 
Change-Id: I77dc71aaf9e14ef743f2a2cbebeec04a4f628c78
 2015-09-14 16:02:10 -07:00
Colin Patrick Mccabe
53bad4eb00 HDFS-8996. Consolidate validateLog and scanLog in FJM#EditLogFile (Zhe Zhang via Colin P. McCabe) 2015-09-14 15:22:12 -07:00
Steve Loughran
6955771261 HDFS-9069. TestNameNodeMetricsLogger failing -port in use. (stevel) 2015-09-14 10:36:44 +01:00
Vinayakumar B
c715650385 HDFS-9036. In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is selected eventhough fallbackToLocalRack is true. (Contributed by J.Andreina) 2015-09-12 17:40:16 +05:30
Vinayakumar B
ce02b5532c HDFS-8975. Erasure coding : Fix random failure in TestSafeModeWithStripedFile (Contributed by J.Andreina) 2015-09-12 13:29:25 +05:30