big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20,640 Commits 2 Branches 2 Tags 264 MiB
ee44b069c6
Commit Graph

204 Commits

Author SHA1 Message Date
Jitendra Pandey
0bc6d0484a HDDS-550. Serialize ApplyTransaction calls per Container in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-10-11 17:12:47 -07:00
Hanisha Koneru
eb34b5f8af HDDS-601. On restart, SCM throws 'No such datanode' exception. 2018-10-11 13:20:32 -07:00
Márton Elek
600438bcde HDDS-602. Bump Ozone version to 0.4.0-SNAPSHOT after ozone-0.3 branch cut. 2018-10-09 19:04:25 +02:00
Nanda kumar
7367ff333b HDDS-450. Generate BlockCommitSequenceId in ContainerStateMachine for every commit operation in Ratis. Contributed by Shashikant Banerjee. 2018-10-09 18:07:01 +05:30
Bharat Viswanadham
2a07617f85 HDDS-354. VolumeInfo.getScmUsed throws NPE. Contributed by Hanisha Koneru. 2018-10-04 21:46:17 -07:00
Nanda kumar
f7ff8c051e HDDS-325. Add event watcher for delete blocks command. Contributed by Lokesh Jain. 2018-10-01 13:50:17 +05:30
Hanisha Koneru
2a5d4315bf HDDS-551. Fix the close container status check in CloseContainerCommandHandler. Contributed by Shashikant Banerjee. 2018-09-27 16:03:49 -07:00
Bharat Viswanadham
62f817d32e HDDS-447. Separate ozone-dist and hadoop-dist projects with real classpath separation. Contributed by Elek Marton. 2018-09-24 10:10:11 -07:00
Anu Engineer
524f7cd354 HDDS-514. Clean Unregister JMX upon SCMConnectionManager#close. 
Contributed by Xiaoyu Yao.
 2018-09-20 12:21:34 -07:00
Anu Engineer
096a716080 HDDS-394. Rename *Key Apis in DatanodeContainerProtocol to *Block apis. 
Contributed Dinesh Chitlangia.
 2018-09-20 11:51:49 -07:00
Nanda kumar
042bf74d5e HDDS-460. Replication manager failed to import container data. Contributed by Elek, Marton. 2018-09-19 23:51:50 +05:30
Nanda kumar
61a4b07bda HDDS-461. Container remains in CLOSING state in SCM forever. Contributed by Shashikant Banerjee. 2018-09-19 21:43:44 +05:30
Márton Elek
b3c5221f30 HDDS-458. numberofKeys is 0 for all containers even when keys are present. Contributed by LiXin Ge. 
(cherry picked from commit 8600b049af184af04dfb93b6bc353740e3e821d3)
 2018-09-19 17:11:05 +02:00
Nanda kumar
c0956ee2a8 HDDS-476. Add Pipeline reports to make pipeline active on SCM restart. 
Contributed by Mukul Kumar Singh.
 2018-09-19 18:52:08 +05:30
Anu Engineer
a968ea4897 HDDS-440. Datanode loops forever if it cannot create directories. 
Contributed by Bharat Viswanadham.
 2018-09-18 14:31:50 -07:00
Anu Engineer
f938925bde HDDS-464. Fix TestCloseContainerHandlingByClient. 
Contributed by Lokesh Jain.
 2018-09-18 10:45:51 -07:00
Nanda kumar
c1df3084ff HDDS-389. Remove XceiverServer and XceiverClient and related classes. Contributed by chencan. 2018-09-15 00:20:19 +05:30
Mukul Kumar Singh
c9e0b69ab3 HDDS-233. Update ozone to latest ratis snapshot build(0.3.0-50588bd-SNAPSHOT). Contributed by Shashikant Banerjee. 2018-09-13 19:01:07 +05:30
Mukul Kumar Singh
c6e19db197 HDDS-420. putKey failing with KEY_ALLOCATION_ERROR. Contributed by Shashikant Banerjee. 2018-09-13 11:29:40 +05:30
Mukul Kumar Singh
47299ce23f HDDS-433. ContainerStateMachine#readStateMachineData should properly build LogEntryProto. Contributed by Lokesh Jain. 2018-09-12 18:31:03 +05:30
Anu Engineer
9c238ffc30 HDDS-432. Replication of closed containers is not working. 
Contributed by Elek, Marton.
 2018-09-11 17:00:04 -07:00
Anu Engineer
7b5886bf78 HDDS-222. Remove hdfs command line from ozone distribution. 
Contributed by Elek, Marton.
 2018-09-11 10:19:50 -07:00
Márton Elek
317f317d4b HDDS-421. Resilient DNS resolution in datanode-service. Contributed by Elek, Marton. 2018-09-10 20:52:52 +02:00
Márton Elek
94ed5cffd7 HDDS-214. Bump ozone/hdds version after the release branch cut 2018-09-07 17:45:26 +02:00
Tsz Wo Nicholas Sze
b3161c4dd9 HDDS-297. Add pipeline actions in Ozone. Contributed by Mukul Kumar Singh and Shashikant Banerjee 2018-09-06 14:35:07 -07:00
Xiaoyu Yao
8286bf2d1f HDDS-303. Removing logic to identify containers to be closed from SCM. Contributed by Nanda kumar. 2018-09-05 12:26:37 -07:00
Nanda kumar
b9932162e9 HDDS-75. Support for CopyContainer. Contributed by Elek, Marton. 2018-09-04 23:41:50 +05:30
Mukul Kumar Singh
2651e2c43d HDDS-365. Implement flushStateMachineData for containerStateMachine. Contributed by Shashikant Banerjee. 2018-08-29 07:58:30 +05:30
Anu Engineer
c5629d546d HDDS-382. Remove RatisTestHelper#RatisTestSuite constructor argument and fix checkstyle in ContainerTestHelper, GenericTestUtils 
Contributed by Nandakumar.
 2018-08-28 14:06:19 -07:00
Nanda kumar
2172399c55 HDDS-381. Fix TestKeys#testPutAndGetKeyWithDnRestart. Contributed by Mukul Kumar Singh. 2018-08-28 22:19:52 +05:30
Márton Elek
a813fd0215 HDDS-227. Use Grpc as the default transport protocol for Standalone pipeline. Contributed by chencan. 2018-08-27 16:07:55 +02:00
Márton Elek
744ce200d2 HDDS-313. Add metrics to containerState Machine. Contributed by chencan. 2018-08-27 15:51:34 +02:00
Nanda kumar
12b2f362cc HDDS-374. Support to configure container size in units lesser than GB. Contributed by Nanda kumar. 2018-08-27 18:29:32 +05:30
Márton Elek
138b0c1443 HDDS-363. Faster datanode registration during the first startup. Contributed by Elek, Marton. 2018-08-24 15:36:10 +02:00
Márton Elek
55b6931059 HDDS-317. Use new StorageSize API for reading ozone.scm.container.size.gb. Contributed by Junjie Chen. 2018-08-24 13:56:30 +02:00
Xiaoyu Yao
ca29fb754e HDDS-328. Support export and import of the KeyValueContainer. Contributed by Elek Marton. 2018-08-23 11:30:28 -07:00
Hanisha Koneru
5aa15cfaff HDDS-265. Move numPendingDeletionBlocks and deleteTransactionId from ContainerData to KeyValueContainerData. Contributed by LiXin Ge. 2018-08-22 10:04:15 -07:00
Mukul Kumar Singh
e3d73bbc24 HDDS-353. Multiple delete Blocks tests are failing consistently. Contributed by Lokesh Jain. 2018-08-20 13:37:58 +05:30
Mukul Kumar Singh
5ef29087ad HDDS-179. CloseContainer/PutKey command should be syncronized with write operations. Contributed by Shashikant Banerjee. 2018-08-16 23:35:19 +05:30
Xiaoyu Yao
b4031a8f1b HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report pipeline info. Contributed by Mukul Kumar Singh. 2018-08-13 12:47:49 -07:00
Mukul Kumar Singh
a8dae0047c HDDS-308. SCM should identify a container with pending deletes using container reports. Contributed by Lokesh Jain. 2018-08-13 12:47:52 +05:30
Hanisha Koneru
a28624d2a4 HDDS-327. CloseContainer command handler should not throw exception if the container is already closed. Contributed by Shashikant Banerjee. 2018-08-10 11:43:22 -07:00
Mukul Kumar Singh
398d895543 HDDS-339. Add block length and blockId in PutKeyResponse. Contributed by Shashikant Banerjee. 2018-08-10 23:45:56 +05:30
Hanisha Koneru
d81cd3611a HDDS-267. Handle consistency issues during container update/close. 2018-08-08 16:47:25 -07:00
Mukul Kumar Singh
900c0e114f HDDS-230. ContainerStateMachine should implement readStateMachineData api to read data from Containers if required during replication. Contributed by Mukul Kumar Singh. 2018-08-07 15:03:14 +05:30
Nanda kumar
db465afb5c HDDS-300. Create a config for volume choosing policy. Contributed by Bharat Viswanadham. 2018-08-03 14:53:04 +05:30
Xiaoyu Yao
40ab8ee597 HDDS-312. Add blockIterator to Container. Contributed by Bharat Viswanadham. 2018-08-02 16:48:21 -07:00
Nanda kumar
41da2050bd HDDS-310. VolumeSet shutdown hook fails on datanode restart. Contributed by Bharat Viswanadham. 2018-08-02 11:35:22 +05:30
Bharat Viswanadham
c835fc08ad HDDS-271. Create a block iterator to iterate blocks in a container. Contributed by Bharat Viswanadham. 2018-07-31 16:26:09 -07:00
Nanda kumar
7631e0adae HDDS-305. Datanode StateContext#addContainerActionIfAbsent will add container action even if there already is a ContainerAction. Contributed by Nanda kumar. 2018-07-31 17:27:51 +05:30
Tsz Wo Nicholas Sze
ee53602a81 HDDS-293. Reduce memory usage and object creation in KeyData. 2018-07-30 15:00:29 -07:00
Nanda kumar
3517a47897 HDDS-287. Add Close ContainerAction to Datanode#StateContext when the container gets full. Contributed by Nanda kumar. 2018-07-30 21:18:42 +05:30
Mukul Kumar Singh
952dc2fd55 HDDS-302. Fix javadoc and add implementation details in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-07-30 18:45:58 +05:30
Bharat Viswanadham
007e6f5113 HDDS-248. Refactor DatanodeContainerProtocol.proto Contributed by Hanisha Koneru. 2018-07-28 14:57:11 -07:00
Mukul Kumar Singh
feb795b58d HDDS-273. DeleteLog entries should be purged only after corresponding DNs commit the transaction. Contributed by Lokesh Jain. 2018-07-29 01:02:24 +05:30
Mukul Kumar Singh
6b038f82da HDDS-246. Datanode should throw BlockNotCommittedException for uncommitted blocks to Ozone Client. Contributed by Shashikant Banerjee. 2018-07-28 22:04:11 +05:30
Anu Engineer
3cc7ce816e HDDS-270. Move generic container util functions to ContianerUtils. 
Contributed by Hanisha Koneru.
 2018-07-27 07:12:21 -07:00
Xiaoyu Yao
d70d845705 HDDS-291. Initialize hadoop metrics system in standalone hdds datanodes. Contributed by Elek Marton. 2018-07-26 13:17:37 -07:00
Nanda kumar
3c4fbc635e HDDS-288. Fix bugs in OpenContainerBlockMap. Contributed by Tsz Wo Nicholas Sze. 2018-07-25 20:27:03 +05:30
Nanda kumar
b507f83e15 HDDS-266. Integrate checksum into .container file. Contributed by Hanisha Koneru. 2018-07-25 18:55:26 +05:30
Mukul Kumar Singh
955f795101 HDDS-203. Add getCommittedBlockLength API in datanode. Contributed by Shashikant Banerjee. 2018-07-25 14:15:54 +05:30
Xiaoyu Yao
cd0b9f1380 HDDS-282. Consolidate logging in scm/container-service. Contributed by Elek Marton. 2018-07-24 10:17:03 -07:00
Mukul Kumar Singh
773d312f74 HDDS-272. TestBlockDeletingService is failing with DiskOutOfSpaceException. Contributed by Lokesh Jain. 2018-07-24 21:23:20 +05:30
Nanda kumar
16f9aee5f5 HDDS-262. Send SCM healthy and failed volumes in the heartbeat. Contributed by Bharat Viswanadham. 2018-07-24 12:09:15 +05:30
Xiaoyu Yao
3a9e25edf5 HDDS-199. Implement ReplicationManager to handle underreplication of closed containers. Contributed by Elek Marton. 2018-07-23 10:28:33 -07:00
Mukul Kumar Singh
bbe2f6225e HDDS-181. CloseContainer should commit all pending open Keys on a datanode. Contributed by Shashikant Banerjee. 2018-07-23 09:13:03 +05:30
Nanda kumar
9fa9e301b0 HDDS-249. Fail if multiple SCM IDs on the DataNode and add SCM ID check after version request. Contributed by Bharat Viswanadham. 2018-07-21 18:46:31 +05:30
Xiaoyu Yao
347c955013 HDDS-260. Support in Datanode for sending ContainerActions to SCM. Contributed by Nanda kumar. 2018-07-20 14:37:13 -07:00
Xiaoyu Yao
d2acf8d560 Revert "HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh." 
This reverts commit 6837121a43231f854b0b22ad20330012439313ce.(Mixed with HDDS-260)
 2018-07-20 14:20:18 -07:00
Xiaoyu Yao
6837121a43 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 13:03:35 -07:00
Bharat Viswanadham
ba25d27ddb HDDS-257. Hook up VolumeSet#shutdown from HddsDispatcher#shutdown. Contributed by Hanisha Koneru 2018-07-20 12:41:52 -07:00
Hanisha Koneru
de894d34f6 HDDS-250. Cleanup ContainerData. 2018-07-20 11:36:42 -07:00
Xiaoyu Yao
68b57ad32c HDDS-259. Implement ContainerReportPublisher and NodeReportPublisher. Contributed by Nanda kumar. 2018-07-20 09:12:48 -07:00
Nanda kumar
e9c44ecfc6 HDDS-269. Refactor IdentifiableEventPayload to use a long ID. Contributed by Ajay Kumar. 2018-07-20 21:39:45 +05:30
Xiaoyu Yao
d5d444732b HDDS-241. Handle Volume in inconsistent state. Contributed by Hanisha Koneru. 2018-07-18 09:38:43 -07:00
Bharat Viswanadham
0927bc4f76 HDDS-251. Integrate BlockDeletingService in KeyValueHandler. Contributed by Lokesh Jain 2018-07-15 10:34:00 -07:00
Nanda kumar
d18507209e HDDS-232. Parallel unit test execution for HDDS/Ozone. Contributed by Arpit Agarwal. 2018-07-13 19:50:52 +05:30
Anu Engineer
f89e265905 HDDS-187. Command status publisher for datanode. 
Contributed by Ajay Kumar.
 2018-07-12 21:35:12 -07:00
Bharat Viswanadham
c275a9a6a0 Merge trunk into HDDS-48 2018-07-09 12:13:03 -07:00
Hanisha Koneru
44e19fc7f7 HDDS-213. Single lock to synchronize KeyValueContainer#update. 2018-07-09 09:33:09 -07:00
Bharat Viswanadham
cb9574a337 HDDS-237. Add updateDeleteTransactionId. Contributed by Bharat Viswanadham and Lokesh Jain 2018-07-06 16:26:40 -07:00
Bharat Viswanadham
e899c4cc01 HDDS-211. Add a create container Lock. Contributed by Bharat Viswanadham 2018-07-06 15:02:40 -07:00
Bharat Viswanadham
7dcf5878a4 HDDS-204. Modify Integration tests for new ContainerIO classes. Contributed by Bharat Viswanadham 2018-07-06 11:01:33 -07:00
Bharat Viswanadham
2c2351e87b HDDS-182:CleanUp Reimplemented classes. Contributed by Hansiha Koneru 2018-07-05 15:23:39 -07:00
Bharat Viswanadham
a404164939 HDDS-229:Remove singleton for Handler. Contributed by Bharat Viswanadham 2018-07-05 11:26:00 -07:00
Nanda kumar
71df8c27c9 HDDS-212. Introduce NodeStateManager to manage the state of Datanodes in SCM. Contributed by Nanda kumar. 2018-07-05 02:11:10 +05:30
Hanisha Koneru
e1f4b3b560 HDDS-176. Add keyCount and container maximum size to ContainerData. Contributed by Bharat Viswanadham. 2018-07-03 09:53:41 -07:00
Hanisha Koneru
44b091a8d7 HDDS-205. Add metrics to HddsDispatcher. Contributed by Bharat Viswanadham. 2018-07-03 09:50:25 -07:00
Bharat Viswanadham
23ebe0f4f0 HDDS-169:Add Volume IO Stats. Contributed by Bharat Viswanadham 2018-06-28 17:10:10 -07:00
Nanda kumar
e4d7227aad HDDS-178: DN should update transactionId on block delete. Contributed by Lokesh Jain. 2018-06-29 04:37:16 +05:30
Bharat Viswanadham
c104525168 Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48 2018-06-28 14:28:23 -07:00
Bharat Viswanadham
52d1d9603e HDDS-183:Integrate Volumeset, ContainerSet and HddsDispatcher. Contributed by Bharat Viswanadham 2018-06-28 14:07:52 -07:00
Hanisha Koneru
13579f9296 HDDS-173. Refactor Dispatcher and implement Handler for new ContainerIO design. 2018-06-28 10:13:30 -07:00
Anu Engineer
18932717c4 HDDS-94. Change ozone datanode command to start the standalone datanode plugin. 
Contributed by  Sandeep Nemuri.
 2018-06-27 14:15:15 -07:00
Xiaoyu Yao
1e30547642 HDDS-170. Fix TestBlockDeletingService#testBlockDeletionTimeout. Contributed by Lokesh Jain. 2018-06-27 13:56:45 -07:00
Xiaoyu Yao
e9ec3d78f5 HDDS-186. Create under replicated queue. Contributed by Ajay Kumar. 2018-06-27 13:35:30 -07:00
Xiaoyu Yao
0d6fe5f36b Revert "HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton" 
This reverts commit aaf03cc459.
 2018-06-27 13:25:45 -07:00
Xiaoyu Yao
aaf03cc459 HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton 2018-06-27 12:39:15 -07:00
Bharat Viswanadham
238fe00ad2 HDDS-192:Create new SCMCommand to request a replication of a container. Contributed by Elek Marton 2018-06-25 21:12:05 -07:00
Anu Engineer
a55d6bba71 HDDS-191. Queue SCMCommands via EventQueue in SCM. 
Contributed by Elek, Marton.
 2018-06-25 13:05:22 -07:00