big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,970 Commits 2 Branches 2 Tags 264 MiB
d1aad44490
Commit Graph

260 Commits

Author SHA1 Message Date
Ajay Kumar
3c798c1e3c HDDS-448. Move NodeStat to NodeStatemanager from SCMNodeManager. Contributed by LiXin Ge. 2018-09-28 10:33:17 -07:00
Ajay Kumar
184544eff8 HDDS-401. Update storage statistics on dead node. Contributed by LiXin Ge. 2018-09-27 09:56:30 -07:00
Ajay Kumar
3928af983e HDDS-370. Add and implement following functions in SCMClientProtocolServer. Contributed by Ajay Kumar. 2018-09-25 13:08:44 -07:00
Bharat Viswanadham
62f817d32e HDDS-447. Separate ozone-dist and hadoop-dist projects with real classpath separation. Contributed by Elek Marton. 2018-09-24 10:10:11 -07:00
Nanda kumar
236d16e3a5 HDDS-507. EventQueue should be shutdown on SCM shutdown. Contributed by Xiaoyu Yao. 2018-09-20 01:22:18 +05:30
Nanda kumar
61a4b07bda HDDS-461. Container remains in CLOSING state in SCM forever. Contributed by Shashikant Banerjee. 2018-09-19 21:43:44 +05:30
Márton Elek
b3c5221f30 HDDS-458. numberofKeys is 0 for all containers even when keys are present. Contributed by LiXin Ge. 
(cherry picked from commit 8600b049af184af04dfb93b6bc353740e3e821d3)
 2018-09-19 17:11:05 +02:00
Nanda kumar
15ed74fa24 HDDS-502. Exception in OM startup when running unit tests. Contributed by Arpit Agarwal. 2018-09-19 19:46:25 +05:30
Nanda kumar
c0956ee2a8 HDDS-476. Add Pipeline reports to make pipeline active on SCM restart. 
Contributed by Mukul Kumar Singh.
 2018-09-19 18:52:08 +05:30
Xiaoyu Yao
e435e12f1f HDDS-506. Fields in AllocateScmBlockResponseProto should be optional. Contributed by Arpit Agarwal. 2018-09-18 23:21:29 -07:00
Xiaoyu Yao
295cce39ed HDDS-366. Update functions impacted by SCM chill mode in StorageContainerLocationProtocol. Contributed by Ajay Kumar. 2018-09-18 13:32:48 -07:00
Anu Engineer
23a6137a40 HDDS-475. Block Allocation returns same BlockID on different keys creation. 
Contributed by Nanda Kumar.
 2018-09-17 14:41:17 -07:00
Nanda kumar
846936698b HDDS-399. Persist open pipeline information across SCM restart. Contributed by Mukul Kumar Singh. 2018-09-17 21:51:54 +05:30
Xiaoyu Yao
95231f1749 HDDS-362. Modify functions impacted by SCM chill mode in ScmBlockLocationProtocol. Contributed by Ajay Kumar. 2018-09-16 17:55:46 -07:00
Nanda kumar
c9fa081897 HDDS-462. Optimize ContainerStateMap#getMatchingContainerIDs in SCM. Contributed by Nanda kumar. 2018-09-15 23:11:39 +05:30
Márton Elek
a65c3ea91c HDDS-449. Add a NULL check to protect DeadNodeHandler#onMessage. Contributed by LiXin Ge. 2018-09-15 13:35:21 +02:00
Nanda kumar
b0a659c8c0 HDDS-466. Handle null in argv of StorageContainerManager#createSCM. Contributed by Nanda kumar 2018-09-15 02:34:33 +05:30
Jitendra Pandey
446cb8301e HDDS-452. 'ozone scm' with incorrect argument first logs all the STARTUP_MSG. Contributed by Namit Maheshwari. 2018-09-14 11:39:36 -07:00
Anu Engineer
0c8a43b9ec HDDS-429. StorageContainerManager lock optimization. 
Contributed by Nanda Kumar.
 2018-09-14 10:08:06 -07:00
Mukul Kumar Singh
c6e19db197 HDDS-420. putKey failing with KEY_ALLOCATION_ERROR. Contributed by Shashikant Banerjee. 2018-09-13 11:29:40 +05:30
Anu Engineer
64c7a12b57 HDDS-436. Allow SCM chill mode to be disabled by configuration. 
Contributed by Ajay Kumar.
 2018-09-12 11:40:24 -07:00
Anu Engineer
9c238ffc30 HDDS-432. Replication of closed containers is not working. 
Contributed by Elek, Marton.
 2018-09-11 17:00:04 -07:00
Anu Engineer
8ffbbf51c9 HDDS-431. LeaseManager of CommandWatcher is not started. 
Contributed by Elek, Marton.
 2018-09-11 10:44:47 -07:00
Anu Engineer
7b5886bf78 HDDS-222. Remove hdfs command line from ozone distribution. 
Contributed by Elek, Marton.
 2018-09-11 10:19:50 -07:00
Hanisha Koneru
ab90248b30 HDDS-400. Check global replication state for containers of dead node. Contributed by Elek, Marton. 2018-09-07 11:20:25 -07:00
Anu Engineer
ff64d35716 HDDS-351. Add chill mode state to SCM. 
Contributed by Ajay Kumar.
 2018-09-07 10:54:35 -07:00
Márton Elek
94ed5cffd7 HDDS-214. Bump ozone/hdds version after the release branch cut 2018-09-07 17:45:26 +02:00
Tsz Wo Nicholas Sze
b3161c4dd9 HDDS-297. Add pipeline actions in Ozone. Contributed by Mukul Kumar Singh and Shashikant Banerjee 2018-09-06 14:35:07 -07:00
Xiaoyu Yao
8286bf2d1f HDDS-303. Removing logic to identify containers to be closed from SCM. Contributed by Nanda kumar. 2018-09-05 12:26:37 -07:00
Xiaoyu Yao
85c3fe341a HDDS-268. Add SCM close container watcher. Contributed by Ajay Kumar. 2018-09-04 22:56:57 -07:00
Hanisha Koneru
9964e33e8d HDDS-369. Remove the containers of a dead node from the container state map. Contributed by Elek, Marton 2018-09-04 14:57:54 -07:00
Nanda kumar
567b5558d5 HDDS-343. Containers are stuck in closing state in scm. Contributed by Elek, Marton. 2018-09-03 20:14:45 +05:30
Anu Engineer
c5629d546d HDDS-382. Remove RatisTestHelper#RatisTestSuite constructor argument and fix checkstyle in ContainerTestHelper, GenericTestUtils 
Contributed by Nandakumar.
 2018-08-28 14:06:19 -07:00
Xiaoyu Yao
c9b63956d9 HDSS-375. ContainerReportHandler should not send replication events for open containers. Contributed by Ajay Kumar. 2018-08-27 10:40:33 -07:00
Márton Elek
a5eba25506 HDDS-364. Update open container replica information in SCM during DN register. Contributed by Ajay Kumar. 2018-08-24 22:42:51 +02:00
Márton Elek
138b0c1443 HDDS-363. Faster datanode registration during the first startup. Contributed by Elek, Marton. 2018-08-24 15:36:10 +02:00
Márton Elek
55b6931059 HDDS-317. Use new StorageSize API for reading ozone.scm.container.size.gb. Contributed by Junjie Chen. 2018-08-24 13:56:30 +02:00
Xiaoyu Yao
4c25f37c6c HDDS-350. ContainerMapping#flushContainerInfo doesn't set containerId. Contributed by Ajay Kumar. 2018-08-22 10:54:10 -07:00
Mukul Kumar Singh
e3d73bbc24 HDDS-353. Multiple delete Blocks tests are failing consistently. Contributed by Lokesh Jain. 2018-08-20 13:37:58 +05:30
Xiaoyu Yao
75fc51588d HDDS-298. Implement SCMClientProtocolServer.getContainerWithPipeline for closed containers. Contributed by Ajay Kumar. 2018-08-14 14:57:46 -07:00
Xiaoyu Yao
b4031a8f1b HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report pipeline info. Contributed by Mukul Kumar Singh. 2018-08-13 12:47:49 -07:00
Mukul Kumar Singh
a8dae0047c HDDS-308. SCM should identify a container with pending deletes using container reports. Contributed by Lokesh Jain. 2018-08-13 12:47:52 +05:30
Xiaoyu Yao
f5dbbfe2e9 HDDS-245. Handle ContainerReports in the SCM. Contributed by Elek Marton. 2018-08-09 16:55:39 -07:00
Mukul Kumar Singh
7c368575a3 HDDS-304. Process ContainerAction from datanode heartbeat in SCM. Contributed by Nanda Kumar. 2018-08-02 17:34:17 +05:30
Mukul Kumar Singh
b28bdc7e8b HDDS-279. DeleteBlocks command should not be sent for open containers. Contributed by Lokesh Jain. 2018-07-31 19:50:40 +05:30
Nanda kumar
3517a47897 HDDS-287. Add Close ContainerAction to Datanode#StateContext when the container gets full. Contributed by Nanda kumar. 2018-07-30 21:18:42 +05:30
Mukul Kumar Singh
feb795b58d HDDS-273. DeleteLog entries should be purged only after corresponding DNs commit the transaction. Contributed by Lokesh Jain. 2018-07-29 01:02:24 +05:30
Xiaoyu Yao
fd31cb6cfe HDDS-277. PipelineStateMachine should handle closure of pipelines in SCM. Contributed by Mukul Kumar Singh. 2018-07-26 13:15:55 -07:00
Nanda kumar
a19229594e HDDS-201. Add name for LeaseManager. Contributed by Sandeep Nemuri. 2018-07-26 19:00:23 +05:30
Xiaoyu Yao
2ced3efe94 HDDS-258. Helper methods to generate NodeReport and ContainerReport for testing. Contributed by Nanda Kumar. 2018-07-23 22:04:20 -07:00
Xiaoyu Yao
3a9e25edf5 HDDS-199. Implement ReplicationManager to handle underreplication of closed containers. Contributed by Elek Marton. 2018-07-23 10:28:33 -07:00
Nanda kumar
9fa9e301b0 HDDS-249. Fail if multiple SCM IDs on the DataNode and add SCM ID check after version request. Contributed by Bharat Viswanadham. 2018-07-21 18:46:31 +05:30
Xiaoyu Yao
9be25e3476 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 14:22:02 -07:00
Xiaoyu Yao
d2acf8d560 Revert "HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh." 
This reverts commit 6837121a43231f854b0b22ad20330012439313ce.(Mixed with HDDS-260)
 2018-07-20 14:20:18 -07:00
Xiaoyu Yao
6837121a43 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 13:03:35 -07:00
Xiaoyu Yao
89a0f80741 HDDS-256. Adding CommandStatusReport Handler. Contributed by Ajay Kumar. 2018-07-20 11:07:09 -07:00
Xiaoyu Yao
68b57ad32c HDDS-259. Implement ContainerReportPublisher and NodeReportPublisher. Contributed by Nanda kumar. 2018-07-20 09:12:48 -07:00
Nanda kumar
1fe5b93843 HDDS-253. SCMBlockDeletingService should publish events for delete blocks to EventQueue. Contributed by Lokesh Jain. 2018-07-13 17:18:42 +05:30
Xiaoyu Yao
3f3f72221f HDDS-238. Add Node2Pipeline Map in SCM to track ratis/standalone pipelines. Contributed by Mukul Kumar Singh. 2018-07-12 22:14:03 -07:00
Anu Engineer
f89e265905 HDDS-187. Command status publisher for datanode. 
Contributed by Ajay Kumar.
 2018-07-12 21:35:12 -07:00
Anu Engineer
556d9b36be HDDS-234. Add SCM node report handler. 
Contributed by Ajay Kumar.
 2018-07-12 12:09:31 -07:00
Anu Engineer
5ee90efed3 HDDS-228. Add the ReplicaMaps to ContainerStateManager. 
Contributed by Ajay Kumar.
 2018-07-12 10:43:24 -07:00
Anu Engineer
632aca5793 HDDS-242. Introduce NEW_NODE, STALE_NODE and DEAD_NODE event 
and corresponding event handlers in SCM.
Contributed by Nanda Kumar.

Recommitting after making sure that patch is clean.
 2018-07-11 12:08:50 -07:00
Anu Engineer
b56785873a Revert "HDDS-242. Introduce NEW_NODE, STALE_NODE and DEAD_NODE event" 
This reverts commit a47ec5dac4.
There was a spurious file in this commit. Revert to clean it.
 2018-07-11 12:03:42 -07:00
Anu Engineer
a47ec5dac4 HDDS-242. Introduce NEW_NODE, STALE_NODE and DEAD_NODE event 
and corresponding event handlers in SCM.
Contributed by Nanda Kumar.
 2018-07-10 15:58:47 -07:00
Bharat Viswanadham
c275a9a6a0 Merge trunk into HDDS-48 2018-07-09 12:13:03 -07:00
Anu Engineer
2f51cd60ef HDDS-217. Move all SCMEvents to a package. 
Contributed by Anu Engineer.
 2018-07-08 11:11:21 -07:00
Arpit Agarwal
061b168529 HDDS-167. Rename KeySpaceManager to OzoneManager. Contributed by Arpit Agarwal. 2018-07-06 12:09:05 -07:00
Nanda kumar
71df8c27c9 HDDS-212. Introduce NodeStateManager to manage the state of Datanodes in SCM. Contributed by Nanda kumar. 2018-07-05 02:11:10 +05:30
Anu Engineer
7ca4f0cefa HDDS-175. Refactor ContainerInfo to remove Pipeline object from it. 
Contributed by Ajay Kumar.
 2018-07-03 14:11:52 -07:00
Bharat Viswanadham
c104525168 Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48 2018-06-28 14:28:23 -07:00
Bharat Viswanadham
52d1d9603e HDDS-183:Integrate Volumeset, ContainerSet and HddsDispatcher. Contributed by Bharat Viswanadham 2018-06-28 14:07:52 -07:00
Anu Engineer
8752a48564 HDDS-193. Make Datanode heartbeat dispatcher in SCM event based. 
Contributed by Elek, Marton.
 2018-06-27 14:18:25 -07:00
Xiaoyu Yao
56a4cdb980 HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton. 2018-06-27 13:28:00 -07:00
Xiaoyu Yao
0d6fe5f36b Revert "HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton" 
This reverts commit aaf03cc459.
 2018-06-27 13:25:45 -07:00
Xiaoyu Yao
aaf03cc459 HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton 2018-06-27 12:39:15 -07:00
Bharat Viswanadham
238fe00ad2 HDDS-192:Create new SCMCommand to request a replication of a container. Contributed by Elek Marton 2018-06-25 21:12:05 -07:00
Anu Engineer
a55d6bba71 HDDS-191. Queue SCMCommands via EventQueue in SCM. 
Contributed by Elek, Marton.
 2018-06-25 13:05:22 -07:00
Mukul Kumar Singh
2a4632d3d7 HDDS-141. Remove PipeLine Class from SCM and move the data field in the Pipeline to ContainerInfo. Contributed by Shashikant Banerjee. 2018-06-17 23:48:49 -07:00
Anu Engineer
ddd09d59f3 HDDS-163. Add Datanode heartbeat dispatcher in SCM. 
Contributed by Nandakumar.
 2018-06-13 20:18:22 -07:00
Anu Engineer
7547740e5c HDDS-161. Add functionality to queue ContainerClose command from SCM Heartbeat Response to Ratis. 
Contributed by Shashikant Banerjee.
 2018-06-13 18:48:59 -07:00
Jitendra Pandey
f4c7c91123 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-13 00:36:02 -07:00
Xiaoyu Yao
23bfd9f7e4 HDDS-72. Add deleteTransactionId field in ContainerInfo. Contributed by Lokesh Jain. 2018-06-11 16:02:32 -07:00
Anu Engineer
78761e87a7 HDDS-127. Add CloseContainerEventHandler in SCM. 
Contributed by Shashikant Banerjee.
 2018-06-07 14:35:22 -07:00
Anu Engineer
3b34148c4f HDDS-88. Create separate message structure to represent ports in DatanodeDetails. 
Contributed by Nanda Kumar.
 2018-05-30 08:52:07 -07:00
Anu Engineer
201440b987 HDDS-81. Moving ContainerReport inside Datanode heartbeat. 
Contributed by Nanda Kumar.
 2018-05-29 12:48:50 -07:00
Anu Engineer
9502b47bd2 HDDS-125. Cleanup HDDS CheckStyle issues. 
Contributed by Anu Engineer.
 2018-05-29 09:54:06 -07:00
Anu Engineer
0cf6e87f92 HDDS-78. Add per volume level storage stats in SCM. 
Contributed by  Shashikant Banerjee.
 2018-05-26 11:11:14 -07:00
Xiaoyu Yao
2d19e7d08f HDDS-80. Remove SendContainerCommand from SCM. Contributed by Nanda Kumar. 2018-05-24 11:10:30 -07:00
Xiaoyu Yao
68c7fd8e60 HDDS-79. Remove ReportState from SCMHeartbeatRequestProto. Contributed by Nanda kumar. 2018-05-22 15:46:59 -07:00
Bharat Viswanadham
3d2d9dbcaa HDDS-87:Fix test failures with uninitialized storageLocation field in storageReport Contributed by Shashikant Banerjee 2018-05-21 08:01:51 -07:00
Mukul Kumar Singh
c97df7712c HDDS-57. TestContainerCloser#testRepeatedClose and TestContainerCloser#testCleanupThreadRuns fail consistently. Contributed by Shashikant Banerjee. 2018-05-21 19:06:26 +05:30
Anu Engineer
7c485a6701 HDDS-38. Add SCMNodeStorage map in SCM class to store storage statistics per Datanode. 
Contributed by Shashikant Banerjee.
 2018-05-17 16:13:28 -07:00
Xiaoyu Yao
59bde09920 HDDS-3. Send NodeReport and ContainerReport when datanodes register. Contributed by Bharat Viswanadham. 2018-05-17 08:52:00 -07:00
Xiaoyu Yao
fc5d49c202 HDDS-29. Fix TestStorageContainerManager#testRpcPermission. Contributed by Mukul Kumar Singh. 2018-05-14 09:10:03 -07:00
Anu Engineer
32cbd0cfd6 HDDS-51. Fix TestDeletedBlockLog#testDeletedBlockTransactions. 
Contributed by Mukul Kumar Singh.
 2018-05-12 13:37:34 -07:00
Anu Engineer
1194ec31d7 HDDS-17. Add node to container map class to simplify state in SCM. 
Contributed by Anu Engineer.
 2018-05-12 09:58:20 -07:00
Anu Engineer
f8b540049d HDDS-42. Inconsistent module names and descriptions. Contributed by Tsz Wo Nicholas Sze. 2018-05-10 11:44:14 -07:00
Mukul Kumar Singh
d72c165161 HDDS-23. Remove SCMNodeAddressList from SCMRegisterRequestProto. Contributed by Nanda Kumar. 2018-05-08 18:11:33 +05:30
Anu Engineer
3a43ac2851 HDDS-1. Remove SCM Block DB. Contributed by Xiaoyu Yao. 2018-05-07 14:58:52 -07:00
Anu Engineer
f0c3dc4cf4 HDDS-13. Refactor StorageContainerManager into seperate RPC endpoints. Contributed by Anu Engineer. 2018-04-30 21:41:10 -07:00
Xiaoyu Yao
20dab05564 HDFS-13407. Ozone: Use separated version schema for Hdds/Ozone projects. Contributed by Elek Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar
17974ba3a6 HDFS-13444. Ozone: Fix checkstyle issues in HDFS-7240. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Nanda kumar
025058f251 HDFS-13446. Ozone: Fix OzoneFileSystem contract test failures. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
8cc13ff962 HDFS-13425. Ozone: Clean-up of ozone related change from hadoop-common-project. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Nanda kumar
aae3ba24ca HDFS-13348. Ozone: Update IP and hostname in Datanode from SCM's response to the register call. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Nanda kumar
12d80f8cae HDFS-13416. Ozone: TestNodeManager tests fail. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Nanda kumar
3779fadc5c HDFS-13324. Ozone: Remove InfoPort and InfoSecurePort from DatanodeDetails. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
82eb5da2b6 HDFS-13342. Ozone: Rename and fix ozone CLI scripts. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
8b832f3c35 HDFS-13405. Ozone: Rename HDSL to HDDS. 
Contributed by Ajay Kumar, Elek Marton, Mukul Kumar Singh, Shashikant Banerjee and Anu Engineer.
 2018-04-26 05:36:04 -07:00