big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,229 Commits 2 Branches 2 Tags 264 MiB
aa9f0850e8
Commit Graph

318 Commits

Author SHA1 Message Date
Márton Elek
dfb518bbf5
HDDS-1397. Avoid the usage of signal handlers in datanodes of the MiniOzoneClusters. 
Closes #701
 2019-04-10 18:35:36 +02:00
Mukul Kumar Singh
df01469141 HDDS-1401. Static ContainerCache in Datanodes can result in overwrite of container db. Contributed by Mukul Kumar Singh. (#708) 2019-04-10 18:00:10 +05:30
Shashikant Banerjee
2d4f6b6daa HDDS-372. There are three buffer copies in BlockOutputStream. Contributed by Shashikant Banerjee. 2019-04-09 10:27:58 +05:30
Doroszlai, Attila
274ab4e86c HDDS-1332. Attempt to fix flaky test testStartStopDatanodeStateMachine (#697) 2019-04-05 13:21:58 -07:00
Yiqun Lin
f96fb05a2b HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka. 2019-04-03 14:01:30 +08:00
Yiqun Lin
d31c86892e HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee. 2019-04-02 16:32:30 +08:00
Xiaoyu Yao
d7a2f9456b HDDS-1350. Fix checkstyle issue in TestDatanodeStateMachine. Contributed by Xiaoyu Yao. (#656) 2019-03-28 14:50:34 -07:00
Arpit Agarwal
b4ed81c4e6 HDDS-1332. Add some logging for flaky test testStartStopDatanodeStateMachine. Contributed by Arpit Agarwal. (#649) 2019-03-27 20:53:30 +05:30
Sandeep Nemuri
5c0a81ad3c HDDS-1310. In datanode once a container becomes unhealthy, datanode restart fails. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-03-26 15:44:18 +05:30
Bharat Viswanadham
128dd91e10
HDDS-1331. In DatanodeStateMachine join check for not null. 2019-03-23 09:43:49 -07:00
Ajay Kumar
771ea6b5e7 Revert "HDDS-1310. In datanode once a container becomes unhealthy, datanode restart fails. Contributed by Sandeep Nemuri." 
This reverts commit efad5717ec.
 2019-03-22 16:20:44 -07:00
Ajay Kumar
efad5717ec HDDS-1310. In datanode once a container becomes unhealthy, datanode restart fails. Contributed by Sandeep Nemuri. 2019-03-22 15:53:12 -07:00
Arpit Agarwal
90afc9ab03
HDDS-1320. Update ozone to latest ratis snapshot build (0.4.0-1fc5ace-SNAPSHOT). Contributed by Mukul Kumar Singh. 2019-03-21 14:24:02 -07:00
Ajay Yadav
f10d493325
HDDS-1119. DN get OM certificate from SCM CA for block token validation. Contributed by Ajay Kumar. (#601) 2019-03-18 23:08:17 -07:00
Bharat Viswanadham
926d548caa
HDDS-1281. Fix the findbug issue caused by HDDS-1163. Contributed by Aravindan Vijayan. 2019-03-15 21:06:44 -07:00
Shashikant Banerjee
155ab6d5d8 HDDS-1098. Introduce Retry Policy in Ozone Client. Contributed by Shashikant Banerjee. 2019-03-15 15:40:59 +05:30
Márton Elek
3d140a537a
HDDS-1247. Bump trunk ozone version to 0.5.0. Contributed by Elek, Marton. 
Closes #588
 2019-03-14 12:01:25 +01:00
Mukul Kumar Singh
ea9fb07fd2 HDDS-1241. Update ozone to latest ratis snapshot build (0.4.0-5680cf5-SNAPSHOT). Contributed by Mukul Kumar Singh. 2019-03-14 15:18:06 +05:30
Mukul Kumar Singh
d602d462a6 HDDS-1256. Fix failure in TestKeyValueContainerCheck. Contributed by Supratim Deka. 2019-03-13 16:22:19 +05:30
Arpit Agarwal
24793d2d97
HDDS-1163. Basic framework for Ozone Data Scrubber. Contributed by Supratim Deka. 2019-03-13 04:32:39 +09:00
Márton Elek
d17e31e062
HDDS-1214. Enable tracing for the datanode read/write path. Contributed by Elek, Marton. 
Closes #550.
 2019-03-12 11:34:56 +01:00
Mukul Kumar Singh
8458ced11e HDDS-1238. Fix Ratis Snapshot creation error if the snapshot file already exists in ContainerStateMachine. Contributed by Shashikant Banerjee. 2019-03-09 22:46:20 +05:30
Ajay Yadav
064f38b3a5 HDDS-594. SCM CA: DN sends CSR and uses certificate issued by SCM. Contributed by Ajay Kumar. (#547) 2019-03-07 14:41:52 -08:00
Lokesh Jain
129fd5dd18 HDDS-1208. ContainerStateMachine should set chunk data as state machine data for ratis. Contributed by Lokesh Jain. 2019-03-06 17:00:37 +05:30
Shashikant Banerjee
62e89dc275 HDDS-1184. Parallelization of write chunks in datanodes is broken. Contributed by Shashikant Banerjee. 2019-03-06 10:00:16 +05:30
Elek, Márton
7f636b47b5 HDDS-919. Enable prometheus endpoints for Ozone datanodes (#502) 
Pull request from Elek, Márton. (#502)
 2019-03-05 12:04:57 -08:00
Shashikant Banerjee
5fcea54a50 HDDS-935. Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart. Contributed by Shashikant Banerjee. 2019-03-05 22:09:46 +05:30
Xiaoyu Yao
9e0f3d1c52 HDDS-1038. Support Service Level Authorization for Ozone. Contributed by Xiaoyu Yao and Ajay Kumar. 2019-02-26 06:02:10 -08:00
Anu Engineer
014e17af78 HDDS-1166. Fix checkstyle line length issues. 
Contributed by Nandakumar.
 2019-02-23 20:31:39 -08:00
Nanda kumar
2bc3cfe28f HDDS-1126. Datanode is trying to qausi-close a container which is already closed. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-02-21 21:37:08 +05:30
Anu Engineer
75e15cc0c4 HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects. 
Contributed by Elek, Marton.
 2019-02-14 23:33:25 -08:00
Márton Elek
e7d1ae52d2 HDDS-1017. Use distributed tracing to indentify performance problems in Ozone. Contributed by Elek, Marton. 2019-02-11 12:18:26 +01:00
Mukul Kumar Singh
214112b2d7 HDDS-1010. ContainerSet#getContainerMap should be renamed. Contributed by Supratim Deka. 2019-02-07 18:06:23 +05:30
Bharat Viswanadham
912d9f790d HDDS-1044. Client doesn't propogate correct error code to client on out of disk space. Contributed by Yiqun Lin. 2019-02-06 20:02:17 -08:00
Yiqun Lin
3efa168e1f HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham. 2019-02-05 10:51:52 +08:00
Arpit Agarwal
c35419579b HDDS-1016. Allow marking containers as unhealthy. Contributed by Arpit Agarwal. 2019-01-30 11:40:50 -08:00
Arpit Agarwal
3b49d7aeae HDDS-989. Check Hdds Volumes for errors. Contributed by Arpit Agarwal. 2019-01-27 11:18:30 -08:00
Márton Elek
2d69a35614 HDDS-982. Fix TestContainerDataYaml#testIncorrectContainerFile. Contributed by Doroszlai, Attila. 2019-01-23 11:31:07 +01:00
Xiaoyu Yao
751bc62df7 Merge branch 'HDDS-4' into trunk 2019-01-18 14:44:24 -08:00
Xiaoyu Yao
2aaaf12f9e HDDS-943. Add block token validation in HddsDispatcher/XceiverServer. Contributed by Ajay Kumar. 2019-01-18 12:11:34 -08:00
Ajay Kumar
01a7f9ed35 HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar. 2019-01-17 10:30:44 -08:00
Bharat Viswanadham
96ea464aaa HDDS-971. ContainerDataConstructor throws exception on QUASI_CLOSED and UNHEALTHY container state. Contributed by Lokesh Jain. 2019-01-16 17:43:30 -08:00
Ajay Kumar
f894d86b2f HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao. 2019-01-16 01:42:08 -07:00
Xiaoyu Yao
7e2770699c HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao. 2019-01-16 01:10:08 -07:00
Ajay Kumar
bb4a26ca32 HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar. 2019-01-16 01:06:59 -07:00
Ajay Kumar
33c274ea2e HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Ajay Kumar
9920506b3d HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao. 2019-01-15 23:01:02 -07:00
Xiaoyu Yao
e47135d9d9 HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar. 2019-01-15 22:53:55 -07:00
Xiaoyu Yao
ff61931f91 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:16:58 -07:00
Márton Elek
32cf0412ce HDDS-965. Ozone: checkstyle improvements and code quality scripts. Contributed by Elek, Marton. 2019-01-08 22:27:19 +01:00
Anu Engineer
ddc0a40507 HDDS-896. Handle over replicated containers in SCM. 
Contributed by Nandakumar.
 2019-01-04 09:04:10 -08:00
Bharat Viswanadham
26e4be7022 HDDS-921. Add JVM pause monitor to Ozone Daemons (OM, SCM and Datanodes). Contributed by Bharat Viswanadham. 2018-12-23 17:33:25 -08:00
Shashikant Banerjee
b51e9e4317 HDDS-912. Update ozone to latest ratis snapshot build (0.4.0-3b0be02-SNAPSHOT). Contributed by Lokesh Jain. 2018-12-18 18:21:17 +05:30
Shashikant Banerjee
ee10ba26dd HDDS-539. Ozone datanode command ignores the invalid options. Contributed by Vinicius Higa Murakami. 2018-12-18 16:10:27 +05:30
Jitendra Pandey
1afba83f2c HDDS-870. Avoid creating block sized buffer in ChunkGroupOutputStream. Contributed by Shashikant Banerjee. 2018-12-08 09:23:10 -08:00
Hanisha Koneru
019836b113 HDDS-858. Start a Standalone Ratis Server on OM 2018-12-06 13:33:52 -08:00
Mukul Kumar Singh
7274115d57 HDDS-890. Handle OverlappingFileLockException during writeStateMachineData in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-12-04 20:12:35 +05:30
Bharat Viswanadham
ef3b03b75a HDDS-885. Fix test failures due to ChecksumData. Contributed by Hanisha Koneru. 2018-12-03 10:52:49 -08:00
Shashikant Banerjee
8f3e12ff07 HDDS-882. Provide a config to optionally turn on/off the sync flag during chunk writes. Contributed by Shashikant Banerjee. 2018-12-02 08:06:24 +05:30
Shashikant Banerjee
5a3c7714c4 HDDS-887. Add DispatcherContext info to Dispatcher from containerStateMachine. Contributed by Shashikant Banerjee. 2018-12-02 08:00:35 +05:30
Yiqun Lin
62f821115b HDDS-886. Unnecessary buffer copy in HddsDispatcher#dispatch. Contributed by Lokesh Jain. 2018-11-30 22:57:29 +08:00
Shashikant Banerjee
5e102f9aa5 HDDS-850. ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-11-29 22:20:08 +05:30
Hanisha Koneru
64a4b6b08b HDDS-284. Checksum for ChunksData. 2018-11-28 10:53:12 -08:00
Márton Elek
ff2484906c HDDS-841. [JDK11] Fix Javadoc errors in hadoop-hdds-container-service module. Contributed by Dinesh Chitlangia. 2018-11-22 12:04:29 +01:00
Shashikant Banerjee
ee44b069c6 HDDS-866. Handle RaftRetryFailureException in OzoneClient. Contributed by Shashikant Banerjee. 2018-11-22 15:02:07 +05:30
Nanda kumar
ebb9245366 HDDS-853. Option to force close a container in Datanode. 
Contributed by Nanda kumar.
 2018-11-22 00:50:27 +05:30
Shashikant Banerjee
f42b2dfdf1 HDDS-865. GrpcXceiverService is added twice to GRPC netty server. Contributed by Xiaoyu Yao. 2018-11-21 23:43:01 +05:30
Shashikant Banerjee
14e1a0a3d6 HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-21 10:51:50 +05:30
Shashikant Banerjee
1734ace35f Revert "HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee." 
This reverts commit c74783036d.
 2018-11-21 01:07:38 +05:30
Shashikant Banerjee
c74783036d HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-20 23:01:02 +05:30
Márton Elek
0123e5197b HDDS-164. Add unit test for HddsDatanodeService. Contributed by Yiqun Lin. 2018-11-20 15:49:48 +01:00
Nanda kumar
e7438a1b38 HDDS-849. Fix NPE in TestKeyValueHandler because of audit log write. 
Contributed by Dinesh Chitlangia.
 2018-11-19 22:47:43 +05:30
Nanda kumar
5a7ca6ac3e HDDS-837. Persist originNodeId as part of .container file in datanode. 
Contributed by Nanda kumar.
 2018-11-19 14:39:55 +05:30
Xiaoyu Yao
29374999b6 HDDS-120. Adding HDDS datanode Audit Log. Contributed by Dinesh Chitlangia. 2018-11-16 14:59:59 -08:00
Nanda kumar
c4d9640028 HDDS-801. Quasi close the container when close is not executed via Ratis. 
Contributed by Nanda kumar.
 2018-11-16 23:11:33 +05:30
Anu Engineer
a16aa2f60b HDDS-825. Code cleanup based on messages from ErrorProne. 
Contributed by Anu Engineer.
 2018-11-15 17:36:09 -08:00
Shashikant Banerjee
b57cc73f83 HDDS-774. Remove OpenContainerBlockMap from datanode. Contributed by Shashikant Banerjee. 2018-11-14 20:05:56 +05:30
Shashikant Banerjee
a948281706 HDDS-834. Datanode goes OOM based because of segment size. Contributed by Mukul Kumar Singh. 2018-11-14 15:53:22 +05:30
Shashikant Banerjee
671fd6524b HDDS-675. Add blocking buffer and use watchApi for flush/close in OzoneClient. Contributed by Shashikant Banerjee. 2018-11-13 23:39:14 +05:30
Jitendra Pandey
f944f33832 HDDS-709. Modify Close Container handling sequence on datanodes. Contributed by Shashikant Banerjee. 2018-11-12 14:08:39 -08:00
Nanda kumar
18fe65d756 HDDS-576. Move ContainerWithPipeline creation to RPC endpoint. 
Contributed by Nanda kumar.
 2018-11-12 23:33:11 +05:30
Nanda kumar
9fe50b4991 HDDS-733. Create container if not exist, as part of chunk write. 
Contributed by Lokesh Jain.
 2018-11-10 00:08:17 +05:30
Shashikant Banerjee
31614bcc7c HDDS-806. Update Ratis to latest snapshot version in ozone. Contributed by Tsz Wo Nicholas Sze and Mukul Kumar Singh. 2018-11-09 00:05:45 +05:30
Nanda kumar
c80f753b0e HDDS-737. Introduce Incremental Container Report. 
Contributed by Nanda kumar.
 2018-11-08 18:33:38 +05:30
Shashikant Banerjee
50f40e0536 HDDS-794. addendum patch to fix compilation failure. Contributed by Shashikant Banerjee. 2018-11-06 00:20:57 +05:30
Arpit Agarwal
5ddefdd507 HDDS-794. Add configs to set StateMachineData write timeout in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-11-05 10:41:28 -08:00
Shashikant Banerjee
942693bddd HDDS-799. Avoid ByteString to byte array conversion cost by using ByteBuffer in Datanode. Contributed by Mukul Kumar Singh. 2018-11-05 23:43:22 +05:30
Arpit Agarwal
c8ca1747c0 HDDS-797. If DN is started before SCM, it does not register. Contributed by Hanisha Koneru. 2018-11-05 09:40:00 -08:00
Yiqun Lin
d16d5f730e HDDS-751. Replace usage of Guava Optional with Java Optional. 2018-11-02 10:50:32 +08:00
Shashikant Banerjee
b13c56742a HDDS-697. update and validate the BCSID for PutSmallFile/GetSmallFile command. Contributed by Shashikant Banerjee. 2018-11-01 10:21:39 +05:30
Arpit Agarwal
08bb0362e0 HDDS-759. Create config settings for SCM and OM DB directories. Contributed by Arpit Agarwal. 2018-10-31 11:23:15 -07:00
Nanda kumar
e4f22b08e0 HDDS-755. ContainerInfo and ContainerReplica protobuf changes. 
Contributed by Nanda kumar.
 2018-10-31 10:29:35 +05:30
Anu Engineer
773f0d1519 HDDS-754. VolumeInfo#getScmUsed throws NPE. 
Contributed by Hanisha Koneru.
 2018-10-30 19:17:57 -07:00
Shashikant Banerjee
7757331dbc HDDS-749. Restructure BlockId class in Ozone. Contributed by Shashikant Banerjee. 2018-10-30 14:15:27 +05:30
Arpit Agarwal
496f0ffe90 HDDS-620. ozone.scm.client.address should be an optional setting. Contributed by chencan and Arpit Agarwal. 2018-10-29 17:14:18 -07:00
Nanda kumar
902345de66 HDDS-728. Datanodes should use different ContainerStateMachine for each pipeline. 
Contributed by Mukul Kumar Singh.
 2018-10-29 19:53:52 +05:30
Nanda kumar
dce4ebe814 HDDS-694. Plugin new Pipeline management code in SCM. 
Contributed by Lokesh Jain.
 2018-10-26 17:55:04 +05:30
Shashikant Banerjee
0891cdda79 HDDS-716. Update ozone to latest ratis snapshot build(0.3.0-aa38160-SNAPSHOT). Contributed by Mukul Kumar Singh. 2018-10-24 16:05:08 +05:30
Shashikant Banerjee
b61846392e HDDS-708. Validate BCSID while reading blocks from containers in datanodes. Contributed by Shashikant Banerjee. 2018-10-23 16:52:17 +05:30
Mukul Kumar Singh
82919a1e7a HDDS-638. Enable ratis snapshots for HDDS datanodes. Contributed by Mukul Kumar Singh. 2018-10-22 19:59:35 +05:30