big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
20,222 Commits 2 Branches 2 Tags 264 MiB
9bbeb52486
Commit Graph

279 Commits

Author SHA1 Message Date
Anu Engineer
6e2129cf4e HDDS-428. OzoneManager lock optimization. 
Contributed by Nanda Kumar.
 2018-09-12 10:38:36 -07:00
Mukul Kumar Singh
47299ce23f HDDS-433. ContainerStateMachine#readStateMachineData should properly build LogEntryProto. Contributed by Lokesh Jain. 2018-09-12 18:31:03 +05:30
Anu Engineer
9c238ffc30 HDDS-432. Replication of closed containers is not working. 
Contributed by Elek, Marton.
 2018-09-11 17:00:04 -07:00
Anu Engineer
a406f6f60e HDDS-424. Consolidate ozone oz parameters to use GNU convention. 
Contributed by Elek, Marton.
 2018-09-11 16:47:33 -07:00
Anu Engineer
8ffbbf51c9 HDDS-431. LeaseManager of CommandWatcher is not started. 
Contributed by Elek, Marton.
 2018-09-11 10:44:47 -07:00
Anu Engineer
7b5886bf78 HDDS-222. Remove hdfs command line from ozone distribution. 
Contributed by Elek, Marton.
 2018-09-11 10:19:50 -07:00
Márton Elek
317f317d4b HDDS-421. Resilient DNS resolution in datanode-service. Contributed by Elek, Marton. 2018-09-10 20:52:52 +02:00
Xiaoyu Yao
8fe4062bfc HDDS-403. Fix createdOn and modifiedOn timestamp for volume, bucket, key. Contributed by Dinesh Chitlangia. 2018-09-10 08:45:49 -07:00
Mukul Kumar Singh
d924ca2e1a HDDS-410. ozone scmcli list is not working properly. Contributed by Hanisha Koneru. 2018-09-09 10:37:30 +05:30
Anu Engineer
73fcbdd296 HDDS-190. Improve shell error message for unrecognized option. 
Contributed by Elek, Marton.
 2018-09-07 12:54:32 -07:00
Hanisha Koneru
ab90248b30 HDDS-400. Check global replication state for containers of dead node. Contributed by Elek, Marton. 2018-09-07 11:20:25 -07:00
Anu Engineer
ff64d35716 HDDS-351. Add chill mode state to SCM. 
Contributed by Ajay Kumar.
 2018-09-07 10:54:35 -07:00
Márton Elek
94ed5cffd7 HDDS-214. Bump ozone/hdds version after the release branch cut 2018-09-07 17:45:26 +02:00
Tsz Wo Nicholas Sze
b3161c4dd9 HDDS-297. Add pipeline actions in Ozone. Contributed by Mukul Kumar Singh and Shashikant Banerjee 2018-09-06 14:35:07 -07:00
Xiaoyu Yao
8286bf2d1f HDDS-303. Removing logic to identify containers to be closed from SCM. Contributed by Nanda kumar. 2018-09-05 12:26:37 -07:00
Xiaoyu Yao
85c3fe341a HDDS-268. Add SCM close container watcher. Contributed by Ajay Kumar. 2018-09-04 22:56:57 -07:00
Anu Engineer
6e4c731471 HDDS-396. Remove openContainers.db from SCM. 
Contributed by Dinesh Chitlangia.
 2018-09-04 16:27:31 -07:00
Hanisha Koneru
9964e33e8d HDDS-369. Remove the containers of a dead node from the container state map. Contributed by Elek, Marton 2018-09-04 14:57:54 -07:00
Nanda kumar
6bbd249011 HDDS-98. Adding Ozone Manager Audit Log. Contributed by Dinesh Chitlangia. 2018-09-05 00:11:07 +05:30
Nanda kumar
b9932162e9 HDDS-75. Support for CopyContainer. Contributed by Elek, Marton. 2018-09-04 23:41:50 +05:30
Nanda kumar
567b5558d5 HDDS-343. Containers are stuck in closing state in scm. Contributed by Elek, Marton. 2018-09-03 20:14:45 +05:30
Mukul Kumar Singh
873ef8ae81 HDDS-263. Add retries in Ozone Client to handle BlockNotCommitted Exception. Contributed by Shashikant Banerjee. 2018-09-03 12:26:34 +05:30
Anu Engineer
ff036e49ff HDDS-357. Use DBStore and TableStore for OzoneManager non-background service. 
Contributed by Nandakumar.
 2018-09-02 11:47:32 -07:00
Nanda kumar
6edf3d2ea3 Revert "HDDS-98. Adding Ozone Manager Audit Log." 
This reverts commit 630b64ec7e.
 2018-09-02 00:18:13 +05:30
Anu Engineer
19abaacdad HDDS-392. Incomplete description about auditMap#key in AuditLogging Framework. 
Contributed by  Dinesh Chitlangia.
 2018-08-31 22:24:30 -07:00
Anu Engineer
76bae4ccb1 HDDS-379. Simplify and improve the cli arg parsing of ozone scmcli. 
Contributed by Elek, Marton.
 2018-08-31 18:11:01 -07:00
Anu Engineer
50d2e3ec41 HDDS-388. Fix the name of the db profile configuration key. 
Contributed by Elek, Marton.
 2018-08-31 14:30:29 -07:00
Anu Engineer
630b64ec7e HDDS-98. Adding Ozone Manager Audit Log. 
Contributed by Dinesh Chitlangia.
 2018-08-31 14:20:56 -07:00
Nanda kumar
0bd4217194 HDDS-380. Remove synchronization from ChunkGroupOutputStream and ChunkOutputStream. Contributed by Shashikant Banerjee. 2018-08-29 13:31:19 +05:30
Mukul Kumar Singh
2651e2c43d HDDS-365. Implement flushStateMachineData for containerStateMachine. Contributed by Shashikant Banerjee. 2018-08-29 07:58:30 +05:30
Anu Engineer
c5629d546d HDDS-382. Remove RatisTestHelper#RatisTestSuite constructor argument and fix checkstyle in ContainerTestHelper, GenericTestUtils 
Contributed by Nandakumar.
 2018-08-28 14:06:19 -07:00
Anu Engineer
ac515d22d8 HDDS-376. Create custom message structure for use in AuditLogging 
Contributed by Dinesh Chitlangia.
 2018-08-28 12:59:08 -07:00
Márton Elek
c61824a189 HDDS-359. RocksDB Profiles support. Contributed by Anu Engineer. 2018-08-28 19:33:13 +02:00
Anu Engineer
df21e1b1dd HDDS-332. Remove the ability to configure ozone.handler.type 
Contributed by Nandakumar and Anu Engineer.
 2018-08-28 09:56:02 -07:00
Nanda kumar
2172399c55 HDDS-381. Fix TestKeys#testPutAndGetKeyWithDnRestart. Contributed by Mukul Kumar Singh. 2018-08-28 22:19:52 +05:30
Mukul Kumar Singh
3974427f67 HDDS-247. Handle CLOSED_CONTAINER_IO exception in ozoneClient. Contributed by Shashikant Banerjee. 2018-08-28 07:12:07 +05:30
Xiaoyu Yao
c9b63956d9 HDSS-375. ContainerReportHandler should not send replication events for open containers. Contributed by Ajay Kumar. 2018-08-27 10:40:33 -07:00
Xiaoyu Yao
6eecd251d8 HDDS-377. Make the ScmClient closable and stop the started threads. Contributed by Elek Marton. 2018-08-27 08:19:38 -07:00
Márton Elek
a813fd0215 HDDS-227. Use Grpc as the default transport protocol for Standalone pipeline. Contributed by chencan. 2018-08-27 16:07:55 +02:00
Márton Elek
744ce200d2 HDDS-313. Add metrics to containerState Machine. Contributed by chencan. 2018-08-27 15:51:34 +02:00
Nanda kumar
12b2f362cc HDDS-374. Support to configure container size in units lesser than GB. Contributed by Nanda kumar. 2018-08-27 18:29:32 +05:30
Márton Elek
a5eba25506 HDDS-364. Update open container replica information in SCM during DN register. Contributed by Ajay Kumar. 2018-08-24 22:42:51 +02:00
Márton Elek
138b0c1443 HDDS-363. Faster datanode registration during the first startup. Contributed by Elek, Marton. 2018-08-24 15:36:10 +02:00
Márton Elek
55b6931059 HDDS-317. Use new StorageSize API for reading ozone.scm.container.size.gb. Contributed by Junjie Chen. 2018-08-24 13:56:30 +02:00
Xiaoyu Yao
ca29fb754e HDDS-328. Support export and import of the KeyValueContainer. Contributed by Elek Marton. 2018-08-23 11:30:28 -07:00
Anu Engineer
b021249ac8 HDDS-356. Support ColumnFamily based RockDBStore and TableStore. 
Contributed by Anu Engineer.
 2018-08-22 18:55:14 -07:00
Xiaoyu Yao
4c25f37c6c HDDS-350. ContainerMapping#flushContainerInfo doesn't set containerId. Contributed by Ajay Kumar. 2018-08-22 10:54:10 -07:00
Hanisha Koneru
5aa15cfaff HDDS-265. Move numPendingDeletionBlocks and deleteTransactionId from ContainerData to KeyValueContainerData. Contributed by LiXin Ge. 2018-08-22 10:04:15 -07:00
Mukul Kumar Singh
e3d73bbc24 HDDS-353. Multiple delete Blocks tests are failing consistently. Contributed by Lokesh Jain. 2018-08-20 13:37:58 +05:30
Mukul Kumar Singh
5ef29087ad HDDS-179. CloseContainer/PutKey command should be syncronized with write operations. Contributed by Shashikant Banerjee. 2018-08-16 23:35:19 +05:30
Xiaoyu Yao
75fc51588d HDDS-298. Implement SCMClientProtocolServer.getContainerWithPipeline for closed containers. Contributed by Ajay Kumar. 2018-08-14 14:57:46 -07:00
Márton Elek
d1830d8b8e HDDS-345. Upgrade RocksDB version from 5.8.0 to 5.14.2. Contributed by Nanda kumar. 2018-08-14 08:33:01 +02:00
Xiaoyu Yao
b4031a8f1b HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report pipeline info. Contributed by Mukul Kumar Singh. 2018-08-13 12:47:49 -07:00
Mukul Kumar Singh
a8dae0047c HDDS-308. SCM should identify a container with pending deletes using container reports. Contributed by Lokesh Jain. 2018-08-13 12:47:52 +05:30
Hanisha Koneru
a28624d2a4 HDDS-327. CloseContainer command handler should not throw exception if the container is already closed. Contributed by Shashikant Banerjee. 2018-08-10 11:43:22 -07:00
Mukul Kumar Singh
398d895543 HDDS-339. Add block length and blockId in PutKeyResponse. Contributed by Shashikant Banerjee. 2018-08-10 23:45:56 +05:30
Xiaoyu Yao
f5dbbfe2e9 HDDS-245. Handle ContainerReports in the SCM. Contributed by Elek Marton. 2018-08-09 16:55:39 -07:00
Márton Elek
778369ea02 HDDS-344. Remove multibyte characters from OzoneAcl. Contributed by Takanobu Asanuma. 2018-08-09 14:26:37 +02:00
Márton Elek
3d96bc6e5f HDDS-219. Genearate version-info.properties for hadoop and ozone. Contributed by Sandeep Nemuri. 2018-08-09 11:06:03 +02:00
Hanisha Koneru
d81cd3611a HDDS-267. Handle consistency issues during container update/close. 2018-08-08 16:47:25 -07:00
Anu Engineer
38784f95fe HDDS-124. Validate all required configs needed for ozone-site.xml and reflect the changes in ozone-default.xml 
Contributed by Dinesh Chitlangia.
 2018-08-07 16:40:33 -07:00
Mukul Kumar Singh
900c0e114f HDDS-230. ContainerStateMachine should implement readStateMachineData api to read data from Containers if required during replication. Contributed by Mukul Kumar Singh. 2018-08-07 15:03:14 +05:30
Nanda kumar
db465afb5c HDDS-300. Create a config for volume choosing policy. Contributed by Bharat Viswanadham. 2018-08-03 14:53:04 +05:30
Xiaoyu Yao
40ab8ee597 HDDS-312. Add blockIterator to Container. Contributed by Bharat Viswanadham. 2018-08-02 16:48:21 -07:00
Mukul Kumar Singh
7c368575a3 HDDS-304. Process ContainerAction from datanode heartbeat in SCM. Contributed by Nanda Kumar. 2018-08-02 17:34:17 +05:30
Nanda kumar
41da2050bd HDDS-310. VolumeSet shutdown hook fails on datanode restart. Contributed by Bharat Viswanadham. 2018-08-02 11:35:22 +05:30
Bharat Viswanadham
c835fc08ad HDDS-271. Create a block iterator to iterate blocks in a container. Contributed by Bharat Viswanadham. 2018-07-31 16:26:09 -07:00
Mukul Kumar Singh
b28bdc7e8b HDDS-279. DeleteBlocks command should not be sent for open containers. Contributed by Lokesh Jain. 2018-07-31 19:50:40 +05:30
Nanda kumar
7631e0adae HDDS-305. Datanode StateContext#addContainerActionIfAbsent will add container action even if there already is a ContainerAction. Contributed by Nanda kumar. 2018-07-31 17:27:51 +05:30
Tsz Wo Nicholas Sze
ee53602a81 HDDS-293. Reduce memory usage and object creation in KeyData. 2018-07-30 15:00:29 -07:00
Nanda kumar
3517a47897 HDDS-287. Add Close ContainerAction to Datanode#StateContext when the container gets full. Contributed by Nanda kumar. 2018-07-30 21:18:42 +05:30
Mukul Kumar Singh
952dc2fd55 HDDS-302. Fix javadoc and add implementation details in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-07-30 18:45:58 +05:30
Bharat Viswanadham
007e6f5113 HDDS-248. Refactor DatanodeContainerProtocol.proto Contributed by Hanisha Koneru. 2018-07-28 14:57:11 -07:00
Mukul Kumar Singh
feb795b58d HDDS-273. DeleteLog entries should be purged only after corresponding DNs commit the transaction. Contributed by Lokesh Jain. 2018-07-29 01:02:24 +05:30
Mukul Kumar Singh
6b038f82da HDDS-246. Datanode should throw BlockNotCommittedException for uncommitted blocks to Ozone Client. Contributed by Shashikant Banerjee. 2018-07-28 22:04:11 +05:30
Anu Engineer
3cc7ce816e HDDS-270. Move generic container util functions to ContianerUtils. 
Contributed by Hanisha Koneru.
 2018-07-27 07:12:21 -07:00
Xiaoyu Yao
d70d845705 HDDS-291. Initialize hadoop metrics system in standalone hdds datanodes. Contributed by Elek Marton. 2018-07-26 13:17:37 -07:00
Xiaoyu Yao
fd31cb6cfe HDDS-277. PipelineStateMachine should handle closure of pipelines in SCM. Contributed by Mukul Kumar Singh. 2018-07-26 13:15:55 -07:00
Nanda kumar
a19229594e HDDS-201. Add name for LeaseManager. Contributed by Sandeep Nemuri. 2018-07-26 19:00:23 +05:30
Nanda kumar
3c4fbc635e HDDS-288. Fix bugs in OpenContainerBlockMap. Contributed by Tsz Wo Nicholas Sze. 2018-07-25 20:27:03 +05:30
Nanda kumar
b507f83e15 HDDS-266. Integrate checksum into .container file. Contributed by Hanisha Koneru. 2018-07-25 18:55:26 +05:30
Nanda kumar
43db0cb518 HDDS-285. Create a generic Metadata Iterator. Contributed by Bharat Viswanadham. 2018-07-25 18:13:49 +05:30
Mukul Kumar Singh
955f795101 HDDS-203. Add getCommittedBlockLength API in datanode. Contributed by Shashikant Banerjee. 2018-07-25 14:15:54 +05:30
Xiaoyu Yao
cd0b9f1380 HDDS-282. Consolidate logging in scm/container-service. Contributed by Elek Marton. 2018-07-24 10:17:03 -07:00
Mukul Kumar Singh
773d312f74 HDDS-272. TestBlockDeletingService is failing with DiskOutOfSpaceException. Contributed by Lokesh Jain. 2018-07-24 21:23:20 +05:30
Nanda kumar
16f9aee5f5 HDDS-262. Send SCM healthy and failed volumes in the heartbeat. Contributed by Bharat Viswanadham. 2018-07-24 12:09:15 +05:30
Xiaoyu Yao
2ced3efe94 HDDS-258. Helper methods to generate NodeReport and ContainerReport for testing. Contributed by Nanda Kumar. 2018-07-23 22:04:20 -07:00
Xiaoyu Yao
3a9e25edf5 HDDS-199. Implement ReplicationManager to handle underreplication of closed containers. Contributed by Elek Marton. 2018-07-23 10:28:33 -07:00
Mukul Kumar Singh
bbe2f6225e HDDS-181. CloseContainer should commit all pending open Keys on a datanode. Contributed by Shashikant Banerjee. 2018-07-23 09:13:03 +05:30
Nanda kumar
9fa9e301b0 HDDS-249. Fail if multiple SCM IDs on the DataNode and add SCM ID check after version request. Contributed by Bharat Viswanadham. 2018-07-21 18:46:31 +05:30
Xiaoyu Yao
347c955013 HDDS-260. Support in Datanode for sending ContainerActions to SCM. Contributed by Nanda kumar. 2018-07-20 14:37:13 -07:00
Xiaoyu Yao
9be25e3476 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 14:22:02 -07:00
Xiaoyu Yao
d2acf8d560 Revert "HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh." 
This reverts commit 6837121a43231f854b0b22ad20330012439313ce.(Mixed with HDDS-260)
 2018-07-20 14:20:18 -07:00
Xiaoyu Yao
6837121a43 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 13:03:35 -07:00
Bharat Viswanadham
ba25d27ddb HDDS-257. Hook up VolumeSet#shutdown from HddsDispatcher#shutdown. Contributed by Hanisha Koneru 2018-07-20 12:41:52 -07:00
Hanisha Koneru
de894d34f6 HDDS-250. Cleanup ContainerData. 2018-07-20 11:36:42 -07:00
Xiaoyu Yao
89a0f80741 HDDS-256. Adding CommandStatusReport Handler. Contributed by Ajay Kumar. 2018-07-20 11:07:09 -07:00
Xiaoyu Yao
68b57ad32c HDDS-259. Implement ContainerReportPublisher and NodeReportPublisher. Contributed by Nanda kumar. 2018-07-20 09:12:48 -07:00
Nanda kumar
e9c44ecfc6 HDDS-269. Refactor IdentifiableEventPayload to use a long ID. Contributed by Ajay Kumar. 2018-07-20 21:39:45 +05:30
Xiaoyu Yao
d5d444732b HDDS-241. Handle Volume in inconsistent state. Contributed by Hanisha Koneru. 2018-07-18 09:38:43 -07:00