big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
18,143 Commits 2 Branches 2 Tags 264 MiB
d64bbf983e
Commit Graph

7011 Commits

Author SHA1 Message Date
James
5ba8e8ab7b Revert HDFS-9448. 2018-03-22 17:19:46 -04:00
James
9a0a34bbfa HDFS-9448. Enable valgrind for libhdfspp unit tests. Contributed by Bob Hansen. 2018-03-22 17:19:45 -04:00
James
584c2a204d HDFS-9452. libhdfs++ Fix memory stomp in OpenFileForRead. Contributed by James Clampffer 2018-03-22 17:19:45 -04:00
James
87362b1c17 HDFS-9117. Config file reader / options classes for libhdfs++. Contributed by Bob Hansen. 2018-03-22 17:19:45 -04:00
James
a38703fdfc HDFS-9359. Test libhdfs++ with existing libhdfs tests. Contributed by Stephen Walkauskas. 2018-03-22 17:19:45 -04:00
James
64f0e78490 HDFS-9368. Implement reads with implicit offset state in libhdfs++. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
6f44d92071 HDFS-9103. Retry reads on DN failure. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
7dd29e548c HDFS-9408. Build both static and dynamic libraries for libhdfspp. Contributed by Stephen Walkauskas. 2018-03-22 17:19:45 -04:00
Haohui Mai
af9bc25ac5 HDFS-9419. Import the optional library into libhdfs++. Contributed by Bob Hansen. 2018-03-22 17:19:45 -04:00
Haohui Mai
a0b8f56317 HDFS-9328. Formalize coding standards for libhdfs++. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
a59714dcdb HDFS-9320. libhdfspp should use sizeof(int32_t) instead of sizeof(int) when parsing data. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
5f1d0957b2 HDFS-9340. libhdfspp fails to compile after HDFS-9207. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
dbd1d0c75b HDFS-8766. Implement a libhdfs(3) compatible API. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
422a6e4ed5 HDFS-9288. Import RapidXML 1.13 for libhdfspp. Contributed by Bob Hansen. 2018-03-22 17:19:45 -04:00
Haohui Mai
a14d1f741f HDFS-9265. InputStreamImpl should hold a shared_ptr of the BlockReader. Contributed by James Clampffer. 2018-03-22 17:19:45 -04:00
Haohui Mai
08794423e8 HDFS-9207. Move the implementation to the hdfs-native-client module. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
9e929a7a0d HDFS-9095. RPC client should fail gracefully when the connection is timed out or reset. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
06326b37c9 HDFS-9108. InputStreamImpl::ReadBlockContinuation stores wrong pointers of buffers. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
6294141aeb HDFS-9116. Suppress false positives from Valgrind on uninitialized variables in tests. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
9480c116ac HDFS-9093. Initialize protobuf fields in RemoteBlockReaderTest. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
a559ef0243 HDFS-9025. Fix compilation issues on arch linux. Contributed by Owen O'Malley. 2018-03-22 17:19:45 -04:00
Haohui Mai
bc47acc9d9 HDFS-8952. InputStream.PositionRead() should be aware of available DNs. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
1efb677976 HDFS-8774. Implement FileSystem and InputStream API for libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
bccc640648 HDFS-8775. SASL support for data transfer protocol in libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
26d9f6cee3 HDFS-8788. Implement unit tests for remote block reader in libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
1bec75a13c HDFS-8764. Generate Hadoop RPC stubs from protobuf definitions. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
65b864f341 HDFS-8759. Implement remote block reader in libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
a6b2fb64c4 HDFS-8758. Implement the continuation library in libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
63eee296c7 HDFS-8745. Use Doxygen to generate documents for libhdfspp. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
40a1f3631d HDFS-8737. Initial implementation of a Hadoop RPC v9 client. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
95b479b8c1 HDFS-8725. Use std::chrono to implement the timer in the asio library. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
eb6c865516 HDFS-8724. Import third_party libraries into the repository. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Haohui Mai
d61e844c20 HDFS-8723. Integrate the build infrastructure with hdfs-client. Contributed by Haohui Mai. 2018-03-22 17:19:45 -04:00
Inigo Goiri
e196d158a2 HDFS-12792. RBF: Test Router-based federation using HDFSContract. Contributed by Inigo Goiri. 2018-03-22 13:32:57 -07:00
Wei-Chiu Chuang
f738d75a86 HDFS-11900. Hedged reads thread pool creation not synchronized. Contributed by John Zhuge. 2018-03-22 11:29:31 -07:00
Kihwal Lee
66461ed98b HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong. 2018-03-22 12:21:10 -05:00
Inigo Goiri
3d0ee8b804 HDFS-13318. RBF: Fix FindBugs in hadoop-hdfs-rbf. Contributed by Ekanth S. 2018-03-22 09:21:52 -07:00
Konstantin V Shvachko
8d898ab25f HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan. 2018-03-21 16:46:03 -07:00
Xiaoyu Yao
389bc6d3da HDFS-11043. TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun. 2018-03-21 13:53:35 -07:00
Yongjun Zhang
6c63cc7d30 HDFS-13315. Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang. 2018-03-20 23:04:05 -07:00
Yiqun Lin
2caba999bb HDFS-13250. RBF: Router to manage requests across multiple subclusters. Contributed by Inigo Goiri. 2018-03-21 11:32:05 +08:00
Yiqun Lin
69fe4407eb HDFS-13307. RBF: Improve the use of setQuota command. Contributed by liuhongtong. 2018-03-21 10:51:35 +08:00
weiy
6e2b5fa493 HDFS-13215. RBF: Move Router to its own module. Contributed by Wei Yan 2018-03-19 22:13:40 -07:00
Chris Douglas
3fc3fa9711 HADOOP-14667. Flexible Visual Studio support. Contributed by Allen Wittenauer 2018-03-19 16:05:55 -07:00
Inigo Goiri
08ff1586d5 HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 2018-03-16 10:29:19 -07:00
Xiaoyu Yao
78b05fde6c HDFS-13280. WebHDFS: Fix NPE in get snasphottable directory list call. Contributed by Lokesh Jain. 2018-03-15 11:02:27 -07:00
Inigo Goiri
e71bc00a47 HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 2018-03-15 10:32:30 -07:00
Xiaoyu Yao
da777a5498 HDFS-13251. Avoid using hard coded datanode data dirs in unit tests(addendum). Contributed by Ajay Kumar. 2018-03-15 10:18:44 -07:00
Inigo Goiri
6de135169e HDFS-12723. TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently. Contributed by Ajay Kumar. 2018-03-15 10:14:35 -07:00
fang zhenyi
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Xiao Chen
5ff22d4c3a HDFS-13246. FileInputStream redundant closes in readReplicasFromCache. Contributed by liaoyuxiangqin. 2018-03-14 21:59:51 -07:00
Yiqun Lin
3a0f4bc0d5 HDFS-13261. Fix incorrect null value check. Contributed by Jianfei Jiang. 2018-03-15 11:06:04 +08:00
Inigo Goiri
41fc7f80be HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router. Contributed by Wei Yan. 2018-03-14 13:14:36 -07:00
Xiaoyu Yao
f83716b7f2 HDFS-13251. Avoid using hard coded datanode data dirs in unit tests. Contributed by Ajay Kumar. 2018-03-14 11:26:58 -07:00
Xiaoyu Yao
0be0f1ce1d HDFS-13275. Adding log for BlockPoolManager#refreshNamenodes failures. Contributed by Ajay Kumar. 2018-03-14 10:38:00 -07:00
Arpit Agarwal
4c57fb0cd9 HDFS-13257. Code cleanup: INode never throws QuotaExceededException. Contributed by Tsz Wo Nicholas Sze. 2018-03-14 09:42:48 -07:00
Xiaoyu Yao
a34d83d2fd HDFS-13249. Document webhdfs support for getting snapshottable directory list. Contributed by Lokesh Jain. 2018-03-14 09:14:26 -07:00
Sammi Chen
ad1b988a82 HDFS-11600. Refactor TestDFSStripedOutputStreamWithFailure test classes. Contributed by Sammi Chen. 2018-03-14 14:39:53 +08:00
Yiqun Lin
76be6cbf6c HDFS-12773. RBF: Improve State Store FS implementation. Contributed by Inigo Goiri. 2018-03-14 11:20:59 +08:00
Arpit Agarwal
9714fc1dd4 HDFS-336. dfsadmin -report should report number of blocks from datanode. Contributed by Bharat Viswanadham. 2018-03-13 16:39:17 -07:00
Arpit Agarwal
39537b7c84 HDFS-13235. DiskBalancer: Update Documentation to add newly added options. Contributed by Bharat Viswanadham. 2018-03-13 16:35:51 -07:00
Xiao Chen
8211a3d469 HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma. 2018-03-13 10:48:45 -07:00
Xiao Chen
f82d38dcb3 Revert "HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma." 
This reverts commit 84c1095586.
 2018-03-13 10:36:16 -07:00
Xiao Chen
3b8dbc2cb7 HDFS-12587. Use Parameterized tests in TestBlockInfoStriped and TestLowRedundancyBlockQueues to test all EC policies. Contributed by Takanobu Asanuma. 2018-03-13 10:14:05 -07:00
Xiao Chen
d6931c30c5 HDFS-13239. Fix non-empty dir warning message when setting default EC policy. Contributed by Bharat Viswanadham. 2018-03-13 10:07:26 -07:00
Xiao Chen
84c1095586 HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma. 2018-03-13 09:58:03 -07:00
Chris Douglas
b2b9ce5859 HDFS-13271. WebHDFS: Add constructor in SnapshottableDirectoryStatus with HdfsFileStatus as argument. Contributed by Lokesh Jain 2018-03-13 09:43:22 -07:00
Xiaoyu Yao
0355ec20eb HDFS-13141. WebHDFS: Add support for getting snasphottable directory list. Contributed by Lokesh Jain. 2018-03-12 20:41:37 -07:00
Chris Douglas
45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Yiqun Lin
19292bc264 HDFS-13226. RBF: Throw the exception if mount table entry validated failed. Contributed by maobaolong. 2018-03-13 11:03:31 +08:00
Yiqun Lin
7fab787de7 HDFS-13253. RBF: Quota management incorrect parent-child relationship judgement. Contributed by Yiqun Lin. 2018-03-13 10:30:20 +08:00
Akira Ajisaka
319defafc1
HDFS-12156. TestFSImage fails without -Pnative 2018-03-13 11:26:56 +09:00
Yiqun Lin
4afd50b106 HDFS-10803. TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space available. Contributed by Yiqun Lin. 2018-03-13 10:15:51 +08:00
Inigo Goiri
91c82c90f0 HDFS-13241. RBF: TestRouterSafemode failed if the port 8888 is in use. Contributed by maobaolong. 2018-03-12 17:28:15 -07:00
Chris Douglas
39a5fbae47 HDFS-12677. Extend TestReconstructStripedFile with a random EC policy. Contributed by Takanobu Asanuma 2018-03-12 14:29:44 -07:00
Chris Douglas
ddb67ca707 HADOOP-14742. Document multi-URI replication Inode for ViewFS. Contributed by Gera Shegalov 2018-03-12 13:43:27 -07:00
Anu Engineer
cceb68ffea HDFS-10618. TestPendingReconstruction#testPendingAndInvalidate is flaky due to race condition. Contributed by Eric Badger. 2018-03-12 12:07:22 -07:00
Inigo Goiri
ac627f561f HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin. 2018-03-12 09:58:56 -07:00
Yiqun Lin
4743d4a2c7 HDFS-13240. RBF: Update some inaccurate document descriptions. Contributed by Yiqun Lin. 2018-03-10 11:28:55 +08:00
Inigo Goiri
8133cd5305 HDFS-13232. RBF: ConnectionPool should return first usable connection. Contributed by Ekanth S. 2018-03-09 18:25:05 -08:00
Inigo Goiri
afe1a3ccd5 HDFS-13212. RBF: Fix router location cache issue. Contributed by Weiwei Wu. 2018-03-09 17:18:51 -08:00
Tsz-Wo Nicholas Sze
ba0da2785d HDFS-13252. Code refactoring: Remove Diff.ListType. 2018-03-09 15:50:26 -08:00
Hanisha Koneru
4eeff62f69 HDFS-13244. Add stack, conf, metrics links to utilities dropdown in NN webUI. Contributed by Bharat Viswanadham. 2018-03-09 15:27:17 -08:00
Lokesh Jain
7b0dc31020
HDFS-13190. Document WebHDFS support for snapshot diff 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Xiaoyu Yao <xyao@apache.org>
 2018-03-09 15:06:15 -08:00
Hanisha Koneru
9a082fbe6e HDFS-11394. Support for getting erasure coding policy through WebHDFS#FileStatus. 2018-03-09 13:20:32 -08:00
Yiqun Lin
122805b43a HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan. 2018-03-09 15:42:57 +08:00
Inigo Goiri
0c2b969e01 HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun. 2018-03-08 09:32:05 -08:00
Hanisha Koneru
19ae4429f9 HDFS-13225. StripeReader#checkMissingBlocks() 's IOException info is incomplete. Contributed by lufei. 2018-03-07 14:17:10 -08:00
Sean Mackrory
46d29e3d7e HDFS-13176. WebHdfs file path gets truncated when having semicolon (;) inside. Contributed by Zsolt Venczel. 2018-03-07 13:33:41 -07:00
Tsz-Wo Nicholas Sze
88fba00caa HDFS-13222. Update getBlocks method to take minBlockSize in RPC calls. Contributed by Bharat Viswanadham 2018-03-07 11:27:53 -08:00
Yiqun Lin
58ea2d7a65 HDFS-13214. RBF: Complete document of Router configuration. Contributed by Yiqun Lin. 2018-03-07 15:26:38 +08:00
Xiaoyu Yao
edf9445708 HDFS-13109. Support fully qualified hdfs path in EZ commands. Contributed by Hanisha Koneru. 2018-03-06 16:44:20 -08:00
Tsz-Wo Nicholas Sze
346caa2095 HDFS-13227. Add a method to calculate cumulative diff over multiple snapshots in DirectoryDiffList. Contributed by Shashikant Banerjee 2018-03-06 13:35:03 -08:00
Tsz-Wo Nicholas Sze
871d0d39fa HDFS-13223. Reduce DiffListBySkipList memory usage. Contributed by Shashikant Banerjee 2018-03-06 12:23:03 -08:00
Xiao Chen
9276ef0665 HDFS-13170. Port webhdfs unmaskedpermission parameter to HTTPFS. Contributed by Stephen O'Donnell. 2018-03-06 09:57:35 -08:00
Inigo Goiri
7060725662 HDFS-13188. Disk Balancer: Support multiple block pools during block move. Contributed by Bharat Viswanadham. 2018-03-06 09:09:32 -08:00
Robert Kanter
21176a8ba7
HADOOP-15282. HADOOP-15235 broke TestHttpFSServerWebServer 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-04 07:33:47 -08:00
Arpit Agarwal
a3c304c5dd HDFS-13178. Disk Balancer: Add a force option to DiskBalancer Execute command. Contributed by Bharat Viswanadham. 2018-03-03 15:42:16 -08:00
Tsz-Wo Nicholas Sze
ba82e5c488 HDFS-13173. Replace ArrayList with DirectoryDiffList(SnapshotSkipList) to store DirectoryDiffs. Contributed by Shashikant Banerjee 2018-03-02 17:47:48 -08:00
Xiao Chen
c75105f07b HDFS-13040. Kerberized inotify client fails despite kinit properly. Contributed by Istvan Fajth, Wei-Chiu Chuang, Xiao Chen. 2018-03-02 16:46:04 -08:00