big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,037 Commits 2 Branches 2 Tags 264 MiB
a9849f65ba
Commit Graph

470 Commits

Author SHA1 Message Date
Anu Engineer
217bdbd940 HDDS-1116.Add java profiler servlet to the Ozone web servers. 
Contributed by Elek, Marton.
 2019-02-15 11:32:06 -08:00
Shashikant Banerjee
de934ba2dc HDDS-1076. TestSCMNodeManager crashed the jvm. Contributed by Lokesh Jain. 2019-02-15 21:13:30 +05:30
Anu Engineer
75e15cc0c4 HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects. 
Contributed by Elek, Marton.
 2019-02-14 23:33:25 -08:00
Anu Engineer
0395f22145 HDDS-1068. Improve the error propagation for ozone sh. 
Contributed by Elek, Marton.
 2019-02-14 22:51:46 -08:00
Bharat Viswanadham
917ac9f108 HDDS-972. Add support for configuring multiple OMs. Contributed by Hanisha Koneru. 2019-02-12 22:07:27 -08:00
Nanda kumar
a536eb5c41 HDDS-360. Use RocksDBStore and TableStore for SCM Metadata. 
Contributed by Anu Engineer.
 2019-02-12 14:26:19 +05:30
Ajay Kumar
d48e61dd36 HDDS-1012. Add Default CertificateClient implementation. Contributed by Ajay Kumar 2019-02-11 18:59:47 -08:00
Márton Elek
e7d1ae52d2 HDDS-1017. Use distributed tracing to indentify performance problems in Ozone. Contributed by Elek, Marton. 2019-02-11 12:18:26 +01:00
Shashikant Banerjee
965d26c9c7 HDDS-1026. Reads should fail over to alternate replica. Contributed by Shashikant Banerjee. 2019-02-10 10:53:16 +05:30
Nanda kumar
0c1bc4dcee HDDS-981. Block allocation should involve pipeline selection and then container selection. 
Contributed by Lokesh Jain.
 2019-02-08 15:44:41 +05:30
Márton Elek
a65aca2fef HDDS-922. Create isolated classloder to use ozonefs with any older hadoop versions. Contributed by Elek, Marton. 2019-02-07 17:02:03 +01:00
Márton Elek
9f2da01591 HDDS-776. Make OM initialization resilient to dns failures. Contributed by Doroszlai, Attila. 2019-02-05 16:11:19 +01:00
Yiqun Lin
3efa168e1f HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham. 2019-02-05 10:51:52 +08:00
Ajay Kumar
9a19d6d48b HDDS-1039. OzoneManager fails to connect with secure SCM. Contributed by Ajay Kumar 2019-02-04 16:30:07 -08:00
Márton Elek
033d97abac HDDS-956. MultipartUpload: List Parts for a Multipart upload key. Contributed by Bharat Viswanadham. 2019-01-31 14:51:31 +01:00
Nanda kumar
45c4cfe790 HDDS-906. Display the ozone version on SCM/OM web ui instead of Hadoop version. 
Contributed by Doroszlai, Attila.
 2019-01-25 12:16:29 +05:30
Xiaoyu Yao
e321b91cb5 HDDS-980. Adding getOMCertificate in SCMSecurityProtocol. Contributed by Ajay Kumar. 2019-01-23 19:32:03 -08:00
Bharat Viswanadham
0ef54f72a6 HDDS-992. ozone-default.xml has invalid text from a stale merge. Contributed by Dinesh Chitlangia. 2019-01-22 13:37:05 -08:00
Xiaoyu Yao
2aaaf12f9e HDDS-943. Add block token validation in HddsDispatcher/XceiverServer. Contributed by Ajay Kumar. 2019-01-18 12:11:34 -08:00
Ajay Kumar
01a7f9ed35 HDDS-597. Ratis: Support secure gRPC endpoint with mTLS for Ratis. Contributed by Ajay Kumar. 2019-01-17 10:30:44 -08:00
Xiaoyu Yao
140565f4db HDDS-967. Fix failures in TestOzoneConfigurationFields. Contributed by Ajay Kumar. 2019-01-16 16:44:33 -07:00
Ajay Kumar
0e16cf1973 HDDS-970. Fix classnotfound error for bouncy castle classes in OM,SCM init. Contributed by Ajay Kumar. 2019-01-16 16:40:05 -07:00
Anu Engineer
a5d0fcf704 HDDS-963. Fix failure in TestOzoneShell due to null check in SecurityConfig. 
Contributed by Ajay Kumar.
 2019-01-16 16:18:39 -07:00
Anu Engineer
1d5734e341 HDDS-938. Add Client APIs for using S3 Auth interface. 
Contributed by  Dinesh Chitlangia.
 2019-01-16 16:18:39 -07:00
Anu Engineer
30bfc9cbd0 HDDS-955. SCM CA: Add CA to SCM. 
Contributed by Anu Engineer.
 2019-01-16 13:43:31 -07:00
Xiaoyu Yao
924bea9730 HDDS-102. SCM CA: SCM CA server signs certificate for approved CSR. Contributed by Anu Engineer. 2019-01-16 13:43:30 -07:00
Ajay Kumar
50c4045fde HDDS-805. Block token: Client api changes for block token. Contributed by Ajay Kumar. 2019-01-16 13:38:27 -07:00
Xiaoyu Yao
2b115222cd HDDS-929. Remove ozone.max.key.len property. Contributed by Ajay Kumar. 2019-01-16 01:42:08 -07:00
Ajay Kumar
f894d86b2f HDDS-115. GRPC: Support secure gRPC endpoint with mTLS. Contributed by Xiaoyu Yao. 2019-01-16 01:42:08 -07:00
Ajay Kumar
6d522dc05d HDDS-804. Block token: Add secret token manager. Contributed by Ajay Kumar. 2019-01-16 01:22:43 -07:00
Anu Engineer
0c8829a9a1 HDDS-696. Bootstrap genesis SCM(CA) with self-signed certificate. 
Contributed by Anu Engineer.
 2019-01-16 01:16:25 -07:00
Xiaoyu Yao
7e2770699c HDDS-9. Add GRPC protocol interceptors for Ozone Block Token. Contributed by Xiaoyu Yao. 2019-01-16 01:10:08 -07:00
Ajay Kumar
bb4a26ca32 HDDS-8. Add OzoneManager Delegation Token support. Contributed by Ajay Kumar. 2019-01-16 01:06:59 -07:00
Xiaoyu Yao
6d6b1a00c2 HDDS-836. Add TokenIdentifier Ozone for delegation token and block token. Contributed by Ajay Kumar. 2019-01-15 23:01:04 -07:00
Anu Engineer
6ad794b1b6 HDDS-778. Add an interface for CA and Clients for Certificate operations 
Contributed by Anu Engineer.
 2019-01-15 23:01:03 -07:00
Ajay Kumar
8b8a3f5435 HDDS-760. Add asf license to TestCertificateSignRequest. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Ajay Kumar
33c274ea2e HDDS-103. SCM CA: Add new security protocol for SCM to expose security related functions. Contributed by Ajay Kumar. 2019-01-15 23:01:03 -07:00
Anu Engineer
c260c19d18 HDDS-101. SCM CA: generate CSR for SCM CA clients. 
Contributed by Xiaoyu Yao.
 2019-01-15 23:01:03 -07:00
Xiaoyu Yao
e89c35a50b HDDS-704. Fix the Dependency convergence issue on HDDS-4. Contributed by Xiaoyu Yao. 2019-01-15 23:01:03 -07:00
Ajay Kumar
8d7c5f4ee7 HDDS-588. SelfSignedCertificate#generateCertificate should sign the certificate the configured security provider. Contributed by Xiaoyu Yao. 2019-01-15 23:01:03 -07:00
Ajay Kumar
9920506b3d HDDS-547. Fix secure docker and configs. Contributed by Xiaoyu Yao. 2019-01-15 23:01:02 -07:00
Ajay Kumar
2d269440b0 HDDS-548. Create a Self-Signed Certificate. Contributed by Anu Engineer. 2019-01-15 22:54:51 -07:00
Ajay Kumar
0b034b7005 HDDS-546. Resolve bouncy castle dependency for hadoop-hdds-common. Contributed by Ajay Kumar. 2019-01-15 22:54:51 -07:00
Xiaoyu Yao
570b503e3e Fix merge conflicts 2019-01-15 22:54:50 -07:00
Xiaoyu Yao
faf53f8262 HDDS-100. SCM CA: generate public/private key pair for SCM/OM/DNs. Contributed by Ajay Kumar. 2019-01-15 22:53:56 -07:00
Xiaoyu Yao
e47135d9d9 HDDS-70. Fix config names for secure ksm and scm. Contributed by Ajay Kumar. 2019-01-15 22:53:55 -07:00
Xiaoyu Yao
bfa4929fdb Revert "Bad merge with 996a627b289947af3894bf83e7b63ec702a665cd" 
This reverts commit 996a627b289947af3894bf83e7b63ec702a665cd.
 2019-01-15 22:40:08 -07:00
Xiaoyu Yao
8e6743eb90 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:40:08 -07:00
Xiaoyu Yao
d3920ecbb7 HDDS-5. Enable OzoneManager kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:40:08 -07:00
Xiaoyu Yao
ff61931f91 HDDS-6. Enable SCM kerberos auth. Contributed by Ajay Kumar. 2019-01-15 22:16:58 -07:00
Márton Elek
32cf0412ce HDDS-965. Ozone: checkstyle improvements and code quality scripts. Contributed by Elek, Marton. 2019-01-08 22:27:19 +01:00
Márton Elek
d14c56d150 HDDS-916. MultipartUpload: Complete Multipart upload request. Contributed by Bharat Viswanadham. 2019-01-07 11:27:36 +01:00
Anu Engineer
999da98d67 HDDS-915. Submit client request to OM Ratis server. 
Contributed by Hanisha Koneru.
 2019-01-04 15:25:26 -08:00
Anu Engineer
0cb3316cec HDDS-482. NullPointer exception thrown on console when cli operation failed. 
Contributed by Nandakumar.
 2019-01-02 11:30:54 -08:00
Arpit Agarwal
cf571133b8 HDDS-893. pipeline status is ALLOCATED in scmcli listPipelines command. Contributed by Lokesh Jain. 2018-12-19 13:42:06 +05:30
Shashikant Banerjee
b51e9e4317 HDDS-912. Update ozone to latest ratis snapshot build (0.4.0-3b0be02-SNAPSHOT). Contributed by Lokesh Jain. 2018-12-18 18:21:17 +05:30
Xiaoyu Yao
94b368f29f HDDS-99. Adding SCM Audit log. Contributed by Dinesh Chitlangia. 2018-12-17 15:40:22 -08:00
Márton Elek
17a8708039 HDDS-879. MultipartUpload: Add InitiateMultipartUpload in ozone. Contributed by Bharat Viswanadham. 2018-12-10 10:05:38 +01:00
Jitendra Pandey
1afba83f2c HDDS-870. Avoid creating block sized buffer in ChunkGroupOutputStream. Contributed by Shashikant Banerjee. 2018-12-08 09:23:10 -08:00
Anu Engineer
5e773efd78 HDDS-909 : Default implementation for Ozone acls. 
Contributed by Ajay Kumar.
 2018-12-07 23:20:19 -08:00
Hanisha Koneru
019836b113 HDDS-858. Start a Standalone Ratis Server on OM 2018-12-06 13:33:52 -08:00
Ajay Kumar
8d882c3786 HDDS-880. Create api for ACL handling in Ozone. (Contributed by Ajay Kumar) 2018-12-06 13:27:28 -08:00
Bharat Viswanadham
343aaea2d1 HDDS-864. Use strongly typed codec implementations for the tables of the OmMetadataManager. Contributed by Elek Marton. 2018-12-06 11:39:59 -08:00
Bharat Viswanadham
ef3b03b75a HDDS-885. Fix test failures due to ChecksumData. Contributed by Hanisha Koneru. 2018-12-03 10:52:49 -08:00
Shashikant Banerjee
8f3e12ff07 HDDS-882. Provide a config to optionally turn on/off the sync flag during chunk writes. Contributed by Shashikant Banerjee. 2018-12-02 08:06:24 +05:30
Shashikant Banerjee
5a3c7714c4 HDDS-887. Add DispatcherContext info to Dispatcher from containerStateMachine. Contributed by Shashikant Banerjee. 2018-12-02 08:00:35 +05:30
Bharat Viswanadham
d15dc43659 HDDS-748. Use strongly typed metadata Table implementation. Contributed by Elek Marton. 2018-12-01 16:52:23 -08:00
Shashikant Banerjee
5e102f9aa5 HDDS-850. ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-11-29 22:20:08 +05:30
Ajay Kumar
184cced513 HDDS-808. Simplify OMAction and DNAction classes used for AuditLogging. Contributed by Dinesh Chitlangia. 2018-11-29 08:35:20 -08:00
Ajay Kumar
b71cc7f33e HDDS-642. Add chill mode exit condition for pipeline availability. Contributed by Yiqun Lin. 2018-11-28 17:47:57 -08:00
Hanisha Koneru
64a4b6b08b HDDS-284. Checksum for ChunksData. 2018-11-28 10:53:12 -08:00
Anu Engineer
34a914be03 HDDS-846. Exports ozone metrics to prometheus. 
Contributed by Elek, Marton.
 2018-11-27 15:57:24 -08:00
Nanda kumar
099e723406 HDDS-868. Handle quasi closed container replicas in SCM. 
Contributed by Nanda kumar.
 2018-11-27 13:01:22 +05:30
Mukul Kumar Singh
f207e30142 HDDS-814. dfs.ratis.leader.election.minimum.timeout.duration should not be read by client. Contributed by chencan. 2018-11-22 10:46:10 +05:30
Ajay Kumar
8d2789c5eb HDDS-795. RocksDb specific classes leak from DBStore/Table interfaces. Contributed by Márton Elek. 2018-11-21 12:25:41 -08:00
Bharat Viswanadham
c4d0ef6946 HDDS-816. Create OM metrics for bucket, volume, keys. Contributed by Bharat Viswanadham. 2018-11-21 10:35:39 -08:00
Shashikant Banerjee
14e1a0a3d6 HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-21 10:51:50 +05:30
Shashikant Banerjee
1734ace35f Revert "HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee." 
This reverts commit c74783036d.
 2018-11-21 01:07:38 +05:30
Shashikant Banerjee
c74783036d HDDS-835. Use storageSize instead of Long for buffer size configs in Ozone Client. Contributed by Shashikant Banerjee. 2018-11-20 23:01:02 +05:30
Mukul Kumar Singh
b5d7b292c9 HDDS-718. Introduce new SCM Commands to list and close Pipelines. Contributed by Lokesh Jain. 2018-11-19 22:59:30 +05:30
Nanda kumar
5a7ca6ac3e HDDS-837. Persist originNodeId as part of .container file in datanode. 
Contributed by Nanda kumar.
 2018-11-19 14:39:55 +05:30
Shashikant Banerjee
10cf5773ba HDDS-845. Create a new raftClient instance for every watch request for Ratis. Contributed by Shashikant Banerjee. 2018-11-19 14:38:51 +05:30
Mukul Kumar Singh
cfb915f3df HDDS-844. Add logic for pipeline teardown after timeout. Contributed by Lokesh Jain. 2018-11-19 12:11:05 +05:30
Xiaoyu Yao
29374999b6 HDDS-120. Adding HDDS datanode Audit Log. Contributed by Dinesh Chitlangia. 2018-11-16 14:59:59 -08:00
Nanda kumar
c4d9640028 HDDS-801. Quasi close the container when close is not executed via Ratis. 
Contributed by Nanda kumar.
 2018-11-16 23:11:33 +05:30
Anu Engineer
a16aa2f60b HDDS-825. Code cleanup based on messages from ErrorProne. 
Contributed by Anu Engineer.
 2018-11-15 17:36:09 -08:00
Giovanni Matteo Fumarola
fcd94eeab8 HDDS-842. [JDK11] Fix Javadoc errors in hadoop-hdds-common module. Contributed by Dinesh Chitlangia. 2018-11-15 14:59:31 -08:00
Márton Elek
2718f7534d HDDS-828. Fix deprecation log generated by getting value of the setting ozone.scm.heartbeat.rpc-timeout. Contributed by Yiqun Lin. 2018-11-15 14:04:55 +01:00
Márton Elek
fb1e8bb0cc HDDS-528. add cli command to checkChill mode status and exit chill mode. Contributed by chencan. 2018-11-15 13:59:56 +01:00
Márton Elek
b12e69475b HDDS-658. Implement s3 bucket list backend call and use it from rest endpoint. Contributed by Bharat Viswanadham. 2018-11-15 12:22:16 +01:00
Shashikant Banerjee
a948281706 HDDS-834. Datanode goes OOM based because of segment size. Contributed by Mukul Kumar Singh. 2018-11-14 15:53:22 +05:30
Shashikant Banerjee
671fd6524b HDDS-675. Add blocking buffer and use watchApi for flush/close in OzoneClient. Contributed by Shashikant Banerjee. 2018-11-13 23:39:14 +05:30
Jitendra Pandey
f944f33832 HDDS-709. Modify Close Container handling sequence on datanodes. Contributed by Shashikant Banerjee. 2018-11-12 14:08:39 -08:00
Nanda kumar
18fe65d756 HDDS-576. Move ContainerWithPipeline creation to RPC endpoint. 
Contributed by Nanda kumar.
 2018-11-12 23:33:11 +05:30
Nanda kumar
9fe50b4991 HDDS-733. Create container if not exist, as part of chunk write. 
Contributed by Lokesh Jain.
 2018-11-10 00:08:17 +05:30
Shashikant Banerjee
31614bcc7c HDDS-806. Update Ratis to latest snapshot version in ozone. Contributed by Tsz Wo Nicholas Sze and Mukul Kumar Singh. 2018-11-09 00:05:45 +05:30
Nanda kumar
c80f753b0e HDDS-737. Introduce Incremental Container Report. 
Contributed by Nanda kumar.
 2018-11-08 18:33:38 +05:30
Akira Ajisaka
49412a1289
HDDS-811. [JDK 11] Fix compilation failures with jdk 11. Contributed by Lokesh Jain. 2018-11-06 21:26:43 +09:00
Arpit Agarwal
5ddefdd507 HDDS-794. Add configs to set StateMachineData write timeout in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-11-05 10:41:28 -08:00
Yiqun Lin
d16d5f730e HDDS-751. Replace usage of Guava Optional with Java Optional. 2018-11-02 10:50:32 +08:00
Shashikant Banerjee
b13c56742a HDDS-697. update and validate the BCSID for PutSmallFile/GetSmallFile command. Contributed by Shashikant Banerjee. 2018-11-01 10:21:39 +05:30
Arpit Agarwal
08bb0362e0 HDDS-759. Create config settings for SCM and OM DB directories. Contributed by Arpit Agarwal. 2018-10-31 11:23:15 -07:00
Nanda kumar
e4f22b08e0 HDDS-755. ContainerInfo and ContainerReplica protobuf changes. 
Contributed by Nanda kumar.
 2018-10-31 10:29:35 +05:30
Shashikant Banerjee
7757331dbc HDDS-749. Restructure BlockId class in Ozone. Contributed by Shashikant Banerjee. 2018-10-30 14:15:27 +05:30
Arpit Agarwal
496f0ffe90 HDDS-620. ozone.scm.client.address should be an optional setting. Contributed by chencan and Arpit Agarwal. 2018-10-29 17:14:18 -07:00
Arpit Agarwal
a58048e8d5 HDDS-727. ozone.log is not getting created in logs directory. Contributed by Nilotpal Nandi. 2018-10-29 09:35:18 -07:00
Nanda kumar
2fa01f823c HDDS-744. Fix ASF license warning in PipelineNotFoundException class. 
Contributed by Lokesh Jain.
 2018-10-27 16:51:08 +05:30
Nanda kumar
dce4ebe814 HDDS-694. Plugin new Pipeline management code in SCM. 
Contributed by Lokesh Jain.
 2018-10-26 17:55:04 +05:30
Hanisha Koneru
38a65e3b7c HDDS-682. Unified o3 address parsing for ozonen sh. Contributed by Elek, Marton. 2018-10-25 17:04:16 -07:00
Arpit Agarwal
244afaba4a HDDS-719. Remove Ozone dependencies on Apache Hadoop 3.2.0. Contributed by Arpit Agarwal. 2018-10-24 16:17:05 -07:00
Shashikant Banerjee
b61846392e HDDS-708. Validate BCSID while reading blocks from containers in datanodes. Contributed by Shashikant Banerjee. 2018-10-23 16:52:17 +05:30
Anu Engineer
e3cca12048 HDDS-676. Enable Read from open Containers via Standalone Protocol. 
Contributed by Shashikant Banerjee.
 2018-10-22 15:57:01 -07:00
Mukul Kumar Singh
82919a1e7a HDDS-638. Enable ratis snapshots for HDDS datanodes. Contributed by Mukul Kumar Singh. 2018-10-22 19:59:35 +05:30
Nanda kumar
c696419f3e Revert "HDDS-705. OS3Exception resource name should be the actual resource name." 
This reverts commit 977c6f6470.

There was spurious edit in this commit.
 2018-10-22 15:37:40 +05:30
Anu Engineer
c9077a9f5a HDDS-707. Allow registering MBeans without additional jmx properties. 
Contributed by Arpit Agarwal.
 2018-10-21 22:50:28 -07:00
Anu Engineer
977c6f6470 HDDS-705. OS3Exception resource name should be the actual resource name. 
Contributed by Bharat Viswanadham.
 2018-10-21 22:49:16 -07:00
Jitendra Pandey
d93d515af5 HDDS-651. Rename o3 to o3fs for Filesystem. 2018-10-17 14:19:17 -07:00
Nandakumar
50715c0699 HDDS-662. Introduce ContainerReplicaState in StorageContainerManager. Contributed by Nanda kumar. 2018-10-17 17:45:35 +05:30
Nandakumar
64a43c92c2 HDDS-656. Add logic for pipeline report and action processing in new pipeline code. Contributed by Lokesh Jain. 2018-10-17 13:57:38 +05:30
Arpit Agarwal
5f0b43fa93 HDDS-378. Remove dependencies between hdds/ozone and hdfs proto files. Contributed by Elek, Marton. 2018-10-16 10:40:00 -07:00
Jitendra Pandey
603649d3a9 HDDS-579. ContainerStateMachine should fail subsequent transactions per container in case one fails. Contributed by Shashikant Banerjee. 2018-10-13 19:15:01 -07:00
Jitendra Pandey
5209c7503b HDDS-603. Add BlockCommitSequenceId field per Container and expose it in Container Reports. Contributed by Shashikant Banerjee. 2018-10-13 12:15:42 -07:00
Jitendra Pandey
22f37af935 Revert "HDDS-629. Make ApplyTransaction calls in ContainerStateMachine idempotent. Contributed by Shashikant Banerjee." 
This reverts commit 0473b68000.
 2018-10-13 12:14:39 -07:00
Jitendra Pandey
0473b68000 HDDS-629. Make ApplyTransaction calls in ContainerStateMachine idempotent. Contributed by Shashikant Banerjee. 2018-10-13 12:12:08 -07:00
Anu Engineer
9227f3d224 HDDS-653. TestMetadataStore#testIterator fails on Windows. 
Contributed by Yiqun Lin.
 2018-10-13 10:41:06 -07:00
Nandakumar
5c8e023ba3 HDDS-587. Add new classes for pipeline management. Contributed by Lokesh Jain. 2018-10-13 03:22:20 +05:30
Arpit Agarwal
3a684a2b23 HDDS-644. Rename dfs.container.ratis.num.container.op.threads. Contributed by Bharat Viswanadham. 2018-10-12 14:44:51 -07:00
Anu Engineer
02e1ef5e07 HDDS-624.PutBlock fails with Unexpected Storage Container Exception. 
Contributed by  Arpit Agarwal.
 2018-10-12 14:22:46 -07:00
Arpit Agarwal
50552479c4 HDDS-636. Turn off ozone.metastore.rocksdb.statistics for now. Contributed by Arpit Agarwal. 2018-10-11 17:21:27 -07:00
Jitendra Pandey
0bc6d0484a HDDS-550. Serialize ApplyTransaction calls per Container in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-10-11 17:12:47 -07:00
Arpit Agarwal
2bd000c851 HDDS-619. hdds.db.profile should not be tagged as a required setting & should default to DISK. Contributed by Dinesh Chitlangia. 2018-10-10 17:37:20 -07:00
Márton Elek
600438bcde HDDS-602. Bump Ozone version to 0.4.0-SNAPSHOT after ozone-0.3 branch cut. 2018-10-09 19:04:25 +02:00
Nanda kumar
9e9915ddab HDDS-599. Fix TestOzoneConfiguration TestOzoneConfigurationFields. Contributed by Sandeep Nemuri. 2018-10-09 18:23:05 +05:30
Nanda kumar
7367ff333b HDDS-450. Generate BlockCommitSequenceId in ContainerStateMachine for every commit operation in Ratis. Contributed by Shashikant Banerjee. 2018-10-09 18:07:01 +05:30
Ajay Kumar
cdf5d58364 HDDS-571. Update SCM chill mode exit criteria to optionally wait for n datanodes. Contributed by Ajay Kumar. 2018-10-05 14:07:16 -07:00
Bharat Viswanadham
2a07617f85 HDDS-354. VolumeInfo.getScmUsed throws NPE. Contributed by Hanisha Koneru. 2018-10-04 21:46:17 -07:00
Anu Engineer
e6b77ad65f HDDS-572. Support S3 buckets as first class objects in Ozone Manager - 1. 
Contributed by Anu Engineer.
 2018-10-04 21:40:13 -07:00
Bharat Viswanadham
4eff629ab3 HDDS-525. Support virtual-hosted style URLs. Contributed by Bharat Viswanadham 2018-10-01 13:16:08 -07:00
Nanda kumar
f7ff8c051e HDDS-325. Add event watcher for delete blocks command. Contributed by Lokesh Jain. 2018-10-01 13:50:17 +05:30
Anu Engineer
0da03f8b14 HDDS-391. Simplify Audit Framework to make audit logging easier to use. 
Contributed by Dinesh Chitlangia.
 2018-09-28 14:00:24 -07:00
Nanda kumar
3f6195045e HDDS-554. In XceiverClientSpi, implement sendCommand(..) using sendCommandAsync(..). Contributed by Tsz Wo Nicholas Sze. 2018-09-26 09:16:23 +05:30
Ajay Kumar
3928af983e HDDS-370. Add and implement following functions in SCMClientProtocolServer. Contributed by Ajay Kumar. 2018-09-25 13:08:44 -07:00
Bharat Viswanadham
9c3fbbc4f6 HDDS-444. Add rest service to the s3gateway. Contributed by Elek Marton. 2018-09-24 22:10:16 -07:00
Bharat Viswanadham
62f817d32e HDDS-447. Separate ozone-dist and hadoop-dist projects with real classpath separation. Contributed by Elek Marton. 2018-09-24 10:10:11 -07:00
Anu Engineer
096a716080 HDDS-394. Rename *Key Apis in DatanodeContainerProtocol to *Block apis. 
Contributed Dinesh Chitlangia.
 2018-09-20 11:51:49 -07:00
Nanda kumar
c0956ee2a8 HDDS-476. Add Pipeline reports to make pipeline active on SCM restart. 
Contributed by Mukul Kumar Singh.
 2018-09-19 18:52:08 +05:30
Xiaoyu Yao
e435e12f1f HDDS-506. Fields in AllocateScmBlockResponseProto should be optional. Contributed by Arpit Agarwal. 2018-09-18 23:21:29 -07:00
Xiaoyu Yao
295cce39ed HDDS-366. Update functions impacted by SCM chill mode in StorageContainerLocationProtocol. Contributed by Ajay Kumar. 2018-09-18 13:32:48 -07:00
Anu Engineer
f938925bde HDDS-464. Fix TestCloseContainerHandlingByClient. 
Contributed by Lokesh Jain.
 2018-09-18 10:45:51 -07:00
Nanda kumar
846936698b HDDS-399. Persist open pipeline information across SCM restart. Contributed by Mukul Kumar Singh. 2018-09-17 21:51:54 +05:30
Xiaoyu Yao
95231f1749 HDDS-362. Modify functions impacted by SCM chill mode in ScmBlockLocationProtocol. Contributed by Ajay Kumar. 2018-09-16 17:55:46 -07:00
Nanda kumar
87e2c0f425 HDDS-465. Suppress group mapping lookup warnings for ozone. Contributed by Xiaoyu Yao. 2018-09-15 23:14:57 +05:30
Nanda kumar
c9fa081897 HDDS-462. Optimize ContainerStateMap#getMatchingContainerIDs in SCM. Contributed by Nanda kumar. 2018-09-15 23:11:39 +05:30
Arpit Agarwal
a12f12f1af HDDS-438. 'ozone oz' should print usage when command or sub-command is missing. Contributed by Dinesh Chitlangia. 2018-09-14 07:02:28 -07:00
Hanisha Koneru
291dcf2247 HDDS-423. Introduce an ozone specific log4j.properties. Contributed by Elek, Marton. 2018-09-13 16:37:44 -07:00
Mukul Kumar Singh
c9e0b69ab3 HDDS-233. Update ozone to latest ratis snapshot build(0.3.0-50588bd-SNAPSHOT). Contributed by Shashikant Banerjee. 2018-09-13 19:01:07 +05:30
Anu Engineer
c18eb97801 HDDS-395. TestOzoneRestWithMiniCluster fails with "Unable to read ROCKDB config". 
Contributed by Dinesh Chitlangia.
 2018-09-12 14:19:01 -07:00
Anu Engineer
64c7a12b57 HDDS-436. Allow SCM chill mode to be disabled by configuration. 
Contributed by Ajay Kumar.
 2018-09-12 11:40:24 -07:00
Anu Engineer
6e2129cf4e HDDS-428. OzoneManager lock optimization. 
Contributed by Nanda Kumar.
 2018-09-12 10:38:36 -07:00
Anu Engineer
a406f6f60e HDDS-424. Consolidate ozone oz parameters to use GNU convention. 
Contributed by Elek, Marton.
 2018-09-11 16:47:33 -07:00
Mukul Kumar Singh
d924ca2e1a HDDS-410. ozone scmcli list is not working properly. Contributed by Hanisha Koneru. 2018-09-09 10:37:30 +05:30
Anu Engineer
73fcbdd296 HDDS-190. Improve shell error message for unrecognized option. 
Contributed by Elek, Marton.
 2018-09-07 12:54:32 -07:00
Anu Engineer
ff64d35716 HDDS-351. Add chill mode state to SCM. 
Contributed by Ajay Kumar.
 2018-09-07 10:54:35 -07:00
Márton Elek
94ed5cffd7 HDDS-214. Bump ozone/hdds version after the release branch cut 2018-09-07 17:45:26 +02:00
Tsz Wo Nicholas Sze
b3161c4dd9 HDDS-297. Add pipeline actions in Ozone. Contributed by Mukul Kumar Singh and Shashikant Banerjee 2018-09-06 14:35:07 -07:00
Xiaoyu Yao
8286bf2d1f HDDS-303. Removing logic to identify containers to be closed from SCM. Contributed by Nanda kumar. 2018-09-05 12:26:37 -07:00
Anu Engineer
6e4c731471 HDDS-396. Remove openContainers.db from SCM. 
Contributed by Dinesh Chitlangia.
 2018-09-04 16:27:31 -07:00
Nanda kumar
6bbd249011 HDDS-98. Adding Ozone Manager Audit Log. Contributed by Dinesh Chitlangia. 2018-09-05 00:11:07 +05:30
Nanda kumar
b9932162e9 HDDS-75. Support for CopyContainer. Contributed by Elek, Marton. 2018-09-04 23:41:50 +05:30
Mukul Kumar Singh
873ef8ae81 HDDS-263. Add retries in Ozone Client to handle BlockNotCommitted Exception. Contributed by Shashikant Banerjee. 2018-09-03 12:26:34 +05:30
Anu Engineer
ff036e49ff HDDS-357. Use DBStore and TableStore for OzoneManager non-background service. 
Contributed by Nandakumar.
 2018-09-02 11:47:32 -07:00
Nanda kumar
6edf3d2ea3 Revert "HDDS-98. Adding Ozone Manager Audit Log." 
This reverts commit 630b64ec7e.
 2018-09-02 00:18:13 +05:30
Anu Engineer
19abaacdad HDDS-392. Incomplete description about auditMap#key in AuditLogging Framework. 
Contributed by  Dinesh Chitlangia.
 2018-08-31 22:24:30 -07:00
Anu Engineer
76bae4ccb1 HDDS-379. Simplify and improve the cli arg parsing of ozone scmcli. 
Contributed by Elek, Marton.
 2018-08-31 18:11:01 -07:00
Anu Engineer
50d2e3ec41 HDDS-388. Fix the name of the db profile configuration key. 
Contributed by Elek, Marton.
 2018-08-31 14:30:29 -07:00
Anu Engineer
630b64ec7e HDDS-98. Adding Ozone Manager Audit Log. 
Contributed by Dinesh Chitlangia.
 2018-08-31 14:20:56 -07:00
Anu Engineer
c5629d546d HDDS-382. Remove RatisTestHelper#RatisTestSuite constructor argument and fix checkstyle in ContainerTestHelper, GenericTestUtils 
Contributed by Nandakumar.
 2018-08-28 14:06:19 -07:00
Anu Engineer
ac515d22d8 HDDS-376. Create custom message structure for use in AuditLogging 
Contributed by Dinesh Chitlangia.
 2018-08-28 12:59:08 -07:00
Márton Elek
c61824a189 HDDS-359. RocksDB Profiles support. Contributed by Anu Engineer. 2018-08-28 19:33:13 +02:00
Anu Engineer
df21e1b1dd HDDS-332. Remove the ability to configure ozone.handler.type 
Contributed by Nandakumar and Anu Engineer.
 2018-08-28 09:56:02 -07:00
Xiaoyu Yao
6eecd251d8 HDDS-377. Make the ScmClient closable and stop the started threads. Contributed by Elek Marton. 2018-08-27 08:19:38 -07:00
Márton Elek
a813fd0215 HDDS-227. Use Grpc as the default transport protocol for Standalone pipeline. Contributed by chencan. 2018-08-27 16:07:55 +02:00
Nanda kumar
12b2f362cc HDDS-374. Support to configure container size in units lesser than GB. Contributed by Nanda kumar. 2018-08-27 18:29:32 +05:30
Márton Elek
55b6931059 HDDS-317. Use new StorageSize API for reading ozone.scm.container.size.gb. Contributed by Junjie Chen. 2018-08-24 13:56:30 +02:00
Anu Engineer
b021249ac8 HDDS-356. Support ColumnFamily based RockDBStore and TableStore. 
Contributed by Anu Engineer.
 2018-08-22 18:55:14 -07:00
Xiaoyu Yao
4c25f37c6c HDDS-350. ContainerMapping#flushContainerInfo doesn't set containerId. Contributed by Ajay Kumar. 2018-08-22 10:54:10 -07:00
Márton Elek
d1830d8b8e HDDS-345. Upgrade RocksDB version from 5.8.0 to 5.14.2. Contributed by Nanda kumar. 2018-08-14 08:33:01 +02:00
Xiaoyu Yao
b4031a8f1b HDDS-324. Use pipeline name as Ratis groupID to allow datanode to report pipeline info. Contributed by Mukul Kumar Singh. 2018-08-13 12:47:49 -07:00
Mukul Kumar Singh
398d895543 HDDS-339. Add block length and blockId in PutKeyResponse. Contributed by Shashikant Banerjee. 2018-08-10 23:45:56 +05:30
Márton Elek
778369ea02 HDDS-344. Remove multibyte characters from OzoneAcl. Contributed by Takanobu Asanuma. 2018-08-09 14:26:37 +02:00
Márton Elek
3d96bc6e5f HDDS-219. Genearate version-info.properties for hadoop and ozone. Contributed by Sandeep Nemuri. 2018-08-09 11:06:03 +02:00
Anu Engineer
38784f95fe HDDS-124. Validate all required configs needed for ozone-site.xml and reflect the changes in ozone-default.xml 
Contributed by Dinesh Chitlangia.
 2018-08-07 16:40:33 -07:00
Nanda kumar
db465afb5c HDDS-300. Create a config for volume choosing policy. Contributed by Bharat Viswanadham. 2018-08-03 14:53:04 +05:30
Bharat Viswanadham
c835fc08ad HDDS-271. Create a block iterator to iterate blocks in a container. Contributed by Bharat Viswanadham. 2018-07-31 16:26:09 -07:00
Tsz Wo Nicholas Sze
ee53602a81 HDDS-293. Reduce memory usage and object creation in KeyData. 2018-07-30 15:00:29 -07:00
Mukul Kumar Singh
952dc2fd55 HDDS-302. Fix javadoc and add implementation details in ContainerStateMachine. Contributed by Shashikant Banerjee. 2018-07-30 18:45:58 +05:30
Bharat Viswanadham
007e6f5113 HDDS-248. Refactor DatanodeContainerProtocol.proto Contributed by Hanisha Koneru. 2018-07-28 14:57:11 -07:00
Mukul Kumar Singh
6b038f82da HDDS-246. Datanode should throw BlockNotCommittedException for uncommitted blocks to Ozone Client. Contributed by Shashikant Banerjee. 2018-07-28 22:04:11 +05:30
Xiaoyu Yao
fd31cb6cfe HDDS-277. PipelineStateMachine should handle closure of pipelines in SCM. Contributed by Mukul Kumar Singh. 2018-07-26 13:15:55 -07:00
Nanda kumar
a19229594e HDDS-201. Add name for LeaseManager. Contributed by Sandeep Nemuri. 2018-07-26 19:00:23 +05:30
Nanda kumar
b507f83e15 HDDS-266. Integrate checksum into .container file. Contributed by Hanisha Koneru. 2018-07-25 18:55:26 +05:30
Nanda kumar
43db0cb518 HDDS-285. Create a generic Metadata Iterator. Contributed by Bharat Viswanadham. 2018-07-25 18:13:49 +05:30
Mukul Kumar Singh
955f795101 HDDS-203. Add getCommittedBlockLength API in datanode. Contributed by Shashikant Banerjee. 2018-07-25 14:15:54 +05:30
Xiaoyu Yao
3a9e25edf5 HDDS-199. Implement ReplicationManager to handle underreplication of closed containers. Contributed by Elek Marton. 2018-07-23 10:28:33 -07:00
Mukul Kumar Singh
bbe2f6225e HDDS-181. CloseContainer should commit all pending open Keys on a datanode. Contributed by Shashikant Banerjee. 2018-07-23 09:13:03 +05:30
Xiaoyu Yao
347c955013 HDDS-260. Support in Datanode for sending ContainerActions to SCM. Contributed by Nanda kumar. 2018-07-20 14:37:13 -07:00
Xiaoyu Yao
9be25e3476 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 14:22:02 -07:00
Xiaoyu Yao
d2acf8d560 Revert "HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh." 
This reverts commit 6837121a43231f854b0b22ad20330012439313ce.(Mixed with HDDS-260)
 2018-07-20 14:20:18 -07:00
Xiaoyu Yao
6837121a43 HDDS-239. Add PipelineStateManager to track pipeline state transition. Contributed by Mukul Kumar Singh. 2018-07-20 13:03:35 -07:00
Hanisha Koneru
de894d34f6 HDDS-250. Cleanup ContainerData. 2018-07-20 11:36:42 -07:00
Xiaoyu Yao
68b57ad32c HDDS-259. Implement ContainerReportPublisher and NodeReportPublisher. Contributed by Nanda kumar. 2018-07-20 09:12:48 -07:00
Bharat Viswanadham
0927bc4f76 HDDS-251. Integrate BlockDeletingService in KeyValueHandler. Contributed by Lokesh Jain 2018-07-15 10:34:00 -07:00
Xiaoyu Yao
3f3f72221f HDDS-238. Add Node2Pipeline Map in SCM to track ratis/standalone pipelines. Contributed by Mukul Kumar Singh. 2018-07-12 22:14:03 -07:00
Anu Engineer
f89e265905 HDDS-187. Command status publisher for datanode. 
Contributed by Ajay Kumar.
 2018-07-12 21:35:12 -07:00
Bharat Viswanadham
c275a9a6a0 Merge trunk into HDDS-48 2018-07-09 12:13:03 -07:00
Anu Engineer
790c563511 HDDS-235. Fix TestOzoneAuditLogger#verifyDefaultLogLevel. 
Contributed by Xiaoyu Yao.
 2018-07-08 11:27:54 -07:00
Arpit Agarwal
061b168529 HDDS-167. Rename KeySpaceManager to OzoneManager. Contributed by Arpit Agarwal. 2018-07-06 12:09:05 -07:00
Bharat Viswanadham
2c2351e87b HDDS-182:CleanUp Reimplemented classes. Contributed by Hansiha Koneru 2018-07-05 15:23:39 -07:00
Nanda kumar
71df8c27c9 HDDS-212. Introduce NodeStateManager to manage the state of Datanodes in SCM. Contributed by Nanda kumar. 2018-07-05 02:11:10 +05:30
Anu Engineer
7ca4f0cefa HDDS-175. Refactor ContainerInfo to remove Pipeline object from it. 
Contributed by Ajay Kumar.
 2018-07-03 14:11:52 -07:00
Anu Engineer
c0ef7e7680 HDDS-198. Create AuditLogger mechanism to be used by OM, SCM and Datanode. 
Contributed by Dinesh Chitlangia.
 2018-07-03 11:39:50 -07:00
Bharat Viswanadham
c104525168 Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48 2018-06-28 14:28:23 -07:00
Bharat Viswanadham
52d1d9603e HDDS-183:Integrate Volumeset, ContainerSet and HddsDispatcher. Contributed by Bharat Viswanadham 2018-06-28 14:07:52 -07:00
Hanisha Koneru
13579f9296 HDDS-173. Refactor Dispatcher and implement Handler for new ContainerIO design. 2018-06-28 10:13:30 -07:00
Xiaoyu Yao
1e30547642 HDDS-170. Fix TestBlockDeletingService#testBlockDeletionTimeout. Contributed by Lokesh Jain. 2018-06-27 13:56:45 -07:00
Xiaoyu Yao
56a4cdb980 HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton. 2018-06-27 13:28:00 -07:00
Xiaoyu Yao
0d6fe5f36b Revert "HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton" 
This reverts commit aaf03cc459.
 2018-06-27 13:25:45 -07:00
Xiaoyu Yao
aaf03cc459 HDDS-194. Remove NodePoolManager and node pool handling from SCM. Contributed by Elek Marton 2018-06-27 12:39:15 -07:00
Mukul Kumar Singh
2a4632d3d7 HDDS-141. Remove PipeLine Class from SCM and move the data field in the Pipeline to ContainerInfo. Contributed by Shashikant Banerjee. 2018-06-17 23:48:49 -07:00
Bharat Viswanadham
998e2850a3 HDDS-155:Implement KeyValueContainer and adopt new disk layout for the containers. Contributed by Bharat Viswanadham 2018-06-14 20:54:54 -07:00
Hanisha Koneru
9a5552bf76 HDDS-156. Implement HDDSVolume to manage volume state 2018-06-14 13:28:41 -07:00
Arpit Agarwal
418cff4820 Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48 2018-06-14 09:38:20 -07:00
Xiaoyu Yao
23bfd9f7e4 HDDS-72. Add deleteTransactionId field in ContainerInfo. Contributed by Lokesh Jain. 2018-06-11 16:02:32 -07:00
Hanisha Koneru
0e437f9b17 Revert "Create Version File in Datanode. Contributed by Bharat Viswanadham." 
This reverts commit f26d3466d7.
 2018-06-11 12:15:39 -07:00
Anu Engineer
a1272448bf HDDS-157. Upgrade common-langs version to 3.7 in HDDS and Ozone. 
Contributed by Takanobu Asanuma.
 2018-06-08 10:27:01 -07:00
Anu Engineer
30f0582f2b HDDS-107. TestOzoneConfigurationFields is failing. 
Contributed by LiXin Ge & Mukul Kumar Singh.
 2018-06-06 08:39:18 -07:00
Bharat Viswanadham
f26d3466d7 Create Version File in Datanode. Contributed by Bharat Viswanadham. 2018-06-04 15:28:19 -07:00
Xiaoyu Yao
ea7b53fb44 HDDS-145. Freon times out because of because of wrong ratis port number in datanode details. Contributed by Mukul Kumar Singh. 2018-06-04 09:51:04 -07:00
Xiaoyu Yao
ab3885f2c8 HDDS-126. Fix findbugs warning in MetadataKeyFilters.java. Contributed by Akira Ajisaka. 2018-06-04 08:55:01 -07:00
Anu Engineer
1be05a3623 HDDS-142. TestMetadataStore fails on Windows. 
Contributed by  Íñigo Goiri.
 2018-06-01 14:21:35 -07:00
Arpit Agarwal
59777185fc HDDS-116. Implement VolumeSet to manage disk volumes. Contributed by Hanisha Koneru. 2018-05-31 10:29:25 -07:00
Bharat Viswanadham
6cd19b45ef HDDS-90: Create ContainerData, Container classes. Contributed by Bharat Viswanadham 2018-05-30 14:04:19 -07:00
Xiaoyu Yao
ee1e0e2036 HDDS-92. Use DBType during parsing datanode .container files. Contributed by Bharat Viswanadham. 2018-05-30 14:04:19 -07:00
Anu Engineer
3b34148c4f HDDS-88. Create separate message structure to represent ports in DatanodeDetails. 
Contributed by Nanda Kumar.
 2018-05-30 08:52:07 -07:00
Anu Engineer
8733012ae3 HDDS-96. Add an option in ozone script to generate a site file with minimally required ozone configs. 
Contributed by Dinesh Chitlangia.
 2018-05-25 13:06:14 -07:00
Mukul Kumar Singh
fed2bef647 HDDS-85. Send Container State Info while sending the container report from Datanode to SCM. Contributed by Shashikant Banerjee. 2018-05-23 14:15:35 +05:30
Anu Engineer
5a9140690a HDDS-49. Standalone protocol should use grpc in place of netty. 
Contributed by Mukul Kumar Singh.
 2018-05-22 19:56:15 -07:00
Anu Engineer
60821fb20e HDDS-74. Rename name of properties related to configuration tags. 
Contributed by Sandeep Nemuri.
 2018-05-22 11:38:11 -07:00
Xiaoyu Yao
5e88126776 HDDS-82. Merge ContainerData and ContainerStatus classes. Contributed by Bharat Viswanadham. 2018-05-21 16:09:24 -07:00
Mukul Kumar Singh
132a547dea HDDS-71. Send ContainerType to Datanode during container creation. Contributed by Bharat Viswanadham. 2018-05-21 23:08:20 +05:30