big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
13,474 Commits 2 Branches 2 Tags 264 MiB
dd3a8bed0a
Commit Graph

214 Commits

Author SHA1 Message Date
Zhe Zhang
0fb14aaced HDFS-9799. Reimplement getCurrentTrashDir to remove incompatibility. (zhz) 
Change-Id: I7834bcebffed38cb384db5395ddb8b6dd9e79a0b
 2016-02-17 13:30:50 -08:00
Haohui Mai
77f7ca3e94 HDFS-9815. Move o.a.h.fs.Hdfs to hadoop-hdfs-client. Contributed by Vinayakumar B. 2016-02-17 10:56:11 -08:00
Jing Zhao
f3c91a41a5 HDFS-9794. Streamer threads may leak if failure happens when closing the striped outputstream. Contributed by Jing Zhao. 2016-02-12 10:59:31 -08:00
Masatake Iwasaki
372d1302c6 HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol (Gary Helmling via iwasakims) 2016-02-13 03:31:15 +09:00
Akira Ajisaka
e6a7044b85 HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. Contributed by Lin Yiqun. 2016-02-13 01:57:24 +09:00
zhezhang
dbe49c1bd6 HDFS-9244. Support nested encryption zones. 
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
 2016-02-08 16:30:51 -08:00
Uma Maheswara Rao G
bffaa38a91 HDFS-9777. Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh) 2016-02-08 13:47:45 -08:00
Kihwal Lee
193d27de0a HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su. 2016-02-08 12:16:29 -06:00
Haohui Mai
1bcfab8e7f HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse ObjectMapper. Contributed by Akira Ajisaka. 2016-02-04 11:34:26 -08:00
Vinayakumar B
d6b1acb940 HDFS-9739. DatanodeStorage.isValidStorageId() is broken (Contributed by Mingliang Liu) 2016-02-03 10:33:33 +05:30
zhezhang
4ae543fdcd HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to BlockECReconstructionCommand. Contributed by Rakesh R. 
Change-Id: I405365a8395770e494b92bfe9651f4f0366d8f28
 2016-02-02 12:32:08 -08:00
Jing Zhao
e30ce01ddc HDFS-9494. Parallel optimization of DFSStripedOutputStream#flushAllInternals. Contributed by Gao Rui. 2016-02-01 13:02:58 -08:00
Tsz-Wo Nicholas Sze
bd909ed9f2 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-26 10:32:51 +08:00
Arpit Agarwal
d62b4a4de7 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:17:05 -08:00
Eric Yang
10a2bc0dff HDFS-9653. Added blocks pending deletion report to dfsadmin. 
(Weiwei Yang via eyang)
 2016-01-24 14:19:49 -08:00
Allen Wittenauer
d22c4239a4 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2016-01-22 12:15:22 -08:00
Jing Zhao
95363bcc7d HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block. Contributed by Jing Zhao. 2016-01-22 09:46:02 -08:00
Kihwal Lee
2fd19b9674 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:10:06 -06:00
Kihwal Lee
3616c7b855 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne. 2016-01-21 09:32:57 -06:00
Colin Patrick Mccabe
7905788db9 HDFS-9576: HTrace: collect position/length information on read operations (zhz via cmccabe) 2016-01-20 11:26:44 -08:00
Kihwal Lee
38c4c14472 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee. 2016-01-08 11:13:58 -06:00
Uma Mahesh
778146eaae HADOOP-12658. Clear javadoc and check style issues around DomainSocket. Contributed by Kai Zheng 2016-01-04 14:32:09 -08:00
Kihwal Lee
867048c3e4 HDFS-7163. WebHdfsFileSystem should retry reads according to the configured retry policy. Contributed by Eric Payne. 2015-12-22 14:08:23 -06:00
Tsz-Wo Nicholas Sze
70d6f20126 HDFS-9575. Use byte array for internal block indices in a striped block. Contributed by jing9 2015-12-22 14:47:57 +08:00
Zhe Zhang
61ab0440f7 HDFS-9173. Erasure Coding: Lease recovery for striped file. Contributed by Walter Su and Jing Zhao. 
Change-Id: I51703a61c9d8454f883028f3f6acb5729fde1b15
 2015-12-18 15:57:48 -08:00
Colin Patrick Mccabe
e63388fdf2 HDFS-9350. Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe) 2015-12-18 09:40:39 -08:00
Zhe Zhang
5104077e1f HDFS-9373. Erasure coding: friendly log information for write operations with some failed streamers. Contributed by Li Bo. 
Change-Id: Ie8ab4ae00e9ee0eb03c32a54bea26a3524308038
 2015-12-17 13:05:04 -08:00
cnauroth
c470c8953d HDFS-9557. Reduce object allocation in PB conversion. Contributed by Daryn Sharp. 2015-12-16 11:10:40 -08:00
Andrew Wang
49949a4bb0 HDFS-8894. Set SO_KEEPALIVE on DN server sockets. Contributed by Kanaka Kumar Avvaru. 2015-12-15 14:38:35 -08:00
Allen Wittenauer
576b569b6c Revert "HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw)" 
This reverts commit 832b3cbde1.
 2015-12-11 10:59:42 -08:00
Allen Wittenauer
832b3cbde1 HDFS-9525. hadoop utilities need to support provided delegation tokens (HeeSoo Kim via aw) 2015-12-08 12:56:38 -08:00
cnauroth
303f80e656 HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. Contributed by Xiaobing Zhou. 2015-12-08 10:49:32 -08:00
Arpit Agarwal
42d49016d4 HDFS-9414. Add missing license header 2015-12-05 09:24:36 -08:00
Arpit Agarwal
86c95cb31a HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. (Contributed by Xiaobing Zhou) 2015-12-04 20:24:08 -08:00
Xiaoyu Yao
cbc7b6bf97 HDFS-8831. Trash Support for deletion in HDFS encryption zone. Contributed by Xiaoyu Yao. 2015-12-04 10:39:45 -08:00
Tsz-Wo Nicholas Sze
e8bd1ba74b HDFS-9294. DFSClient deadlock when close file and failed to renew lease. Contributed by Brahma Reddy Battula 2015-12-02 17:39:28 -08:00
Xiaoyu Yao
e3d673901b HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info. Contributed by Xiaoyu Yao. 2015-11-25 13:41:06 -08:00
Uma Mahesh
411e2b2e7c HDFS-9433. DFS getEZForPath API on a non-existent file should throw FileNotFoundException (Rakesh R via umamahesh) 2015-11-23 17:54:31 -08:00
Haohui Mai
298a8cb096 HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. Contributed by Walter Su. 2015-11-23 10:50:12 -08:00
Haohui Mai
6039059c37 HDFS-9443. Disabling HDFS client socket cache causes logging message printed to console for CLI commands. Contributed by Chris Nauroth. 2015-11-22 15:03:35 -08:00
Akira Ajisaka
ac1aa6c819 HDFS-9431. DistributedFileSystem#concat fails if the target path is relative. Contributed by Kazuho Fujii. 2015-11-19 20:55:01 +09:00
Zhe Zhang
73b94d7899 HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. Contributed by Xiao Chen. 
Change-Id: I9e42f724f27924cf73891425a832de37ce014a1e
 2015-11-10 09:55:29 -08:00
Xiaoyu Yao
41d3f8899d HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. Contributed by Surendra Singh Lilhore. 2015-11-09 09:57:56 -08:00
Kihwal Lee
ef926b2e38 HDFS-9383. TestByteArrayManager#testByteArrayManager fails. Contributed by Tsz Wo Nicholas Sze. 2015-11-09 08:56:04 -06:00
Vinayakumar B
6d2332ae37 HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula) 2015-11-08 16:24:27 +05:30
Tsz-Wo Nicholas Sze
9614dea71a HDFS-9398. Make ByteArraryManager log message in one-line format. Contributed by Mingliang Liu 2015-11-07 22:21:28 +08:00
cnauroth
66c0967310 HDFS-9384. TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server. Contributed by Chris Nauroth. 2015-11-05 22:07:08 -08:00
cnauroth
c563401796 HDFS-9378. hadoop-hdfs-client tests do not write logs. Contributed by Chris Nauroth. 2015-11-05 06:47:58 -08:00
Zhe Zhang
dac0463a4e HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. Contributed by Chang Li. 
Change-Id: If5ce1b2d212bb0726bce52ad12a3de401bcec02d
 2015-11-03 13:34:24 -08:00
Walter Su
6e4f8a46c5 HADOOP-12047. Indicate preference not to affect input buffers during coding in erasure coder. (Contributed by Kai Zheng) 2015-11-02 10:40:14 +08:00
Walter Su
5eca6dece6 HADOOP-12327. Initialize output buffers with ZERO bytes in erasure coder. Contributed by Kai Zheng. 2015-10-29 15:04:34 +08:00
Uma Mahesh
07ecdb877d HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group. (Rakesh R via umamahesh) 2015-10-28 23:33:11 -07:00
Haohui Mai
65f53f246a HDFS-9168. Move client side unit test to hadoop-hdfs-client. Contributed by Haohui Mai. 2015-10-28 15:54:37 -07:00
yliu
c201cf951d HADOOP-12040. Adjust inputs order for the decode API in raw erasure coder. (Kai Zheng via yliu) 2015-10-28 16:18:23 +08:00
Ming Ma
aa09880ab8 HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario. (Mingliang Liu via mingma) 2015-10-27 09:28:40 -07:00
Yongjun Zhang
5e718de522 HDFS-7284. Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. (Wei-Chiu Chuang via Yongjun Zhang) 2015-10-26 13:25:31 -07:00
Arpit Agarwal
86c92227fc HDFS-4015. Safemode should count and report orphaned blocks. (Contributed by Anu Engineer) 2015-10-23 18:07:17 -07:00
Haohui Mai
15eb84b37e HDFS-9301. HDFS clients can't construct HdfsConfiguration instances. Contributed by Mingliang Liu. 2015-10-23 14:53:34 -07:00
Kihwal Lee
b9e0417bdf HDFS-9290. DFSClient#callAppend() is not backward compatible for slightly older NameNodes. Contributed by Tony Wu. 2015-10-23 16:37:45 -05:00
Andrew Wang
8175c4f6b9 HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. Contributed by Xiao Chen. 2015-10-19 16:12:56 -07:00
cnauroth
4f6e842ba9 HDFS-9085. Show renewer information in DelegationTokenIdentifier#toString. Contributed by Zhihai Xu. 2015-10-09 14:27:13 -07:00
Andrew Wang
c32614f410 HDFS-8941. DistributedFileSystem listCorruptFileBlocks API should resolve relative path. Contributed by Rakesh R. 2015-10-09 11:57:03 -07:00
Akira Ajisaka
b4390d5ab8 HDFS-9217. Fix broken findbugsExcludeFile.xml for hadoop-hdfs-client module. Contributed by Mingliang Liu. 2015-10-09 10:23:07 +09:00
Andrew Wang
66e2cfa1a0 HDFS-8632. Add InterfaceAudience annotation to the erasure coding classes. Contributed by Rakesh R. 2015-10-07 18:12:26 -07:00
Jing Zhao
239d119c67 HDFS-9196. Fix TestWebHdfsContentLength. Contributed by Masatake Iwasaki. 2015-10-06 23:54:49 -07:00
Uma Mahesh
8b7339312c HDFS-9182. Cleanup the findbugs and other issues after HDFS EC merged to trunk. Contributed by Uma Maheswara Rao G. 2015-10-06 22:22:32 -07:00
Jing Zhao
a8b4d0ff28 HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers. Contributed by Jing Zhao. 2015-10-06 10:56:10 -07:00
Vinayakumar B
874c8ed239 HDFS-7899. Improve EOF error message (Contributed by Jagadesh Kiran N) 2015-10-06 15:42:36 +05:30
Haohui Mai
7136e8c558 HDFS-8979. Clean up checkstyle warnings in hadoop-hdfs-client module. Contributed by Mingliang Liu. 2015-10-03 11:38:45 -07:00
Jing Zhao
c6cafc77e6 HDFS-9185. Fix null tracer in ErasureCodingWorker. Contributed by Rakesh R. 2015-10-02 11:08:17 -07:00
cnauroth
c7e03c3a6c HDFS-9175. Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public. Contributed by Santhosh Nayak. 2015-09-30 16:06:34 -07:00
Zhe Zhang
6c17d31528 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java

Change-Id: I53ec1c426dc988d6c4a2c87b00caef49c4057010
 2015-09-30 08:39:13 -07:00
Haohui Mai
39285e6a19 HDFS-8971. Remove guards when calling LOG.debug() and LOG.trace() in client package. Contributed by Mingliang Liu. 2015-09-29 17:52:36 -07:00
Haohui Mai
6ee0539ede Revert "HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. Contributed by Haohui Mai." 
This reverts commit d5a9a3daa0.
 2015-09-29 17:51:15 -07:00
Haohui Mai
d5a9a3daa0 HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. Contributed by Haohui Mai. 2015-09-29 17:48:29 -07:00
Haohui Mai
80d33b589b HDFS-9165. Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client. Contributed by Mingliang Liu. 2015-09-29 09:50:04 -07:00
Colin Patrick Mccabe
850d679acb HADOOP-12447. Clean up some htrace integration issues (cmccabe) 2015-09-29 09:25:11 -07:00
Zhe Zhang
8fd5520246 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java

Change-Id: Ic7946c4ea35bed587fe879ce58b959b25ecc0823
 2015-09-29 01:39:16 -07:00
Kihwal Lee
3abbdc929b HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 2015-09-28 17:31:51 -05:00
Jing Zhao
6419900ac2 HDFS-9040. Erasure coding: coordinate data streamers in DFSStripedOutputStream. Contributed by Jing Zhao and Walter Su. 2015-09-28 14:40:27 -07:00
Lei Xu
e5992ef4df HDFS-9147. Fix the setting of visibleLength in ExternalBlockReader. (Colin P. McCabe via Lei (Eddy) Xu) 2015-09-28 13:39:00 -07:00
Kihwal Lee
4c9497cbf0 HDFS-9106. Transfer failure during pipeline recovery causes permanent write failures. Contributed by Kihwal Lee. 2015-09-28 13:29:56 -05:00
Colin Patrick Mccabe
892ade689f HDFS-9080. Update htrace version to 4.0.1 (cmccabe) 2015-09-28 07:42:12 -07:00
Haohui Mai
1c030c6e58 HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-27 10:54:44 -07:00
Haohui Mai
bf37d3d80e HDFS-8053. Move DFSIn/OutputStream and related classes to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-26 11:08:25 -07:00
Lei Xu
67b0e967f0 HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) 2015-09-25 16:01:41 -07:00
Lei Xu
5eb237d544 HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via Lei (Eddy) Xu) 2015-09-25 15:50:19 -07:00
Yongjun Zhang
83e65c5fe8 HDFS-5795. RemoteBlockReader2#checkSuccess() shoud print error status. (Xiao Chen via Yongjun Zhang) 2015-09-24 22:13:14 -07:00
Haohui Mai
692b1a45ce HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from HdfsServerConstants to HdfsConstants. Contributed by Mingliang Liu. 2015-09-24 20:19:16 -07:00
Haohui Mai
ead1b9e680 HDFS-9131. Move config keys used by hdfs-client to HdfsClientConfigKeys. Contributed by Mingliang Liu. 2015-09-24 00:30:01 -07:00
Zhe Zhang
c09dc258a8 HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from ErasureCodingPolicy. Contributed by Vinayakumar B. 
Change-Id: Ic56da0b426f47c63dac440aef6f5fc8554f6cf13
 2015-09-23 13:34:09 -07:00
Haohui Mai
7c5c099324 HDFS-8733. Keep server related definition in hdfs.proto on server side. Contributed by Mingliang Liu. 2015-09-22 20:57:05 -07:00
Haohui Mai
63d9f1596c HDFS-9039. Separate client and server side methods of o.a.h.hdfs.NameNodeProxies. Contributed by Mingliang Liu. 2015-09-22 20:52:37 -07:00
Zhe Zhang
1080c37300 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java

Change-Id: I8511c4d64b0959e79129febc179845a3892fedcc
 2015-09-22 13:27:09 -07:00
Haohui Mai
06022b8fdc HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to PBHelperClient. Contributed by Mingliang Liu. 2015-09-21 18:53:13 -07:00
Lei Xu
3a9c7076e8 HDFS-9004. Add upgrade domain to DatanodeInfo. Contributed by Ming Ma (via Lei (Eddy) Xu). 
Change-Id: I887c66578eebd61acc34b94f18da6e6851c609f4
 2015-09-19 18:08:09 -07:00
Haohui Mai
9eee97508f HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-17 14:18:00 -07:00
cnauroth
559c09dc0e HDFS-9082. Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG. Contributed by Santhosh Nayak. 2015-09-15 15:13:43 -07:00
Haohui Mai
0c7d3f4805 HDFS-9041. Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client. Contributed by Mingliang Liu. 2015-09-12 11:02:51 -07:00
Arpit Agarwal
15a557fcfe HDFS-9027. Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method. (Contributed by Mingliang Liu) 2015-09-11 10:11:52 -07:00