big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,637 Commits 2 Branches 2 Tags 264 MiB
312d4d4cc5
Commit Graph

697 Commits

Author SHA1 Message Date
Ajay Yadav
f10d493325
HDDS-1119. DN get OM certificate from SCM CA for block token validation. Contributed by Ajay Kumar. (#601) 2019-03-18 23:08:17 -07:00
Bharat Viswanadham
93db5da4d9
HDDS-1250. In OM HA AllocateBlock call where connecting to SCM from OM should not happen on Ratis. 2019-03-18 13:51:44 -07:00
Xiaoyu Yao
66a104bc57 HDDS-1296. Fix checkstyle issue from Nightly run. Contributed by Xiaoyu Yao. (#616) 2019-03-18 08:38:55 -07:00
Sammi Chen
4d2a116d6e HDDS-699. Detect Ozone Network topology. Contributed by Sammi Chen. 2019-03-17 17:08:29 +08:00
Xiaoyu Yao
5cfb88a225 HDDS-1246. Add ozone delegation token utility subcmd for Ozone CLI. Contributed by Xiaoyu Yao. (#594) 2019-03-15 16:08:30 -07:00
Elek, Márton
16b78622cc HDDS-1283. Fix the dynamic documentation of basic s3 client usage. Contributed by Elek, Marton. (#605) 2019-03-15 14:12:10 -07:00
Xiaoyu Yao
d1afa03804
HDDS-1138. Ozone Client should avoid talking to SCM directly. Contributed by Xiaoyu Yao and Mukul Kumar Singh. 
Closes #585
 2019-03-15 10:41:06 -07:00
Elek, Márton
dc21655f2a HDDS-1259. OzoneFS classpath separation is broken by the token validation. Contributed by Elek Marton. 
Closes #604
 2019-03-15 09:43:01 -07:00
Shashikant Banerjee
a7f5e742a6 HDDS-1088. Add blockade Tests to test Replica Manager. Contributed by Nilotpal Nandi. 2019-03-15 20:54:41 +05:30
Shashikant Banerjee
155ab6d5d8 HDDS-1098. Introduce Retry Policy in Ozone Client. Contributed by Shashikant Banerjee. 2019-03-15 15:40:59 +05:30
Vivek Ratnavel Subramanian
9001508485 HDDS-761. Create S3 subcommand to run S3 related operations. 
Closes #579
 2019-03-14 21:16:59 -07:00
Vivek Ratnavel Subramanian
2627dad333 HDDS-1265. ozone sh s3 getsecret throws Null Pointer Exception for unsecured clusters. 
Closes #611
 2019-03-14 17:21:06 -07:00
Bharat Viswanadham
091a664977
HDDS-917. Expose NodeManagerMXBean as a MetricsSource. Contributed by Siddharth Wagle. 2019-03-14 12:41:52 -07:00
Mukul Kumar Singh
983b78afdc HDDS-1237. Fix test TestSecureContainerServer.testClientServerRatisGrpc. Contributed by Mukul Kumar Singh. 2019-03-14 16:49:43 +05:30
Márton Elek
3d140a537a
HDDS-1247. Bump trunk ozone version to 0.5.0. Contributed by Elek, Marton. 
Closes #588
 2019-03-14 12:01:25 +01:00
Mukul Kumar Singh
ea9fb07fd2 HDDS-1241. Update ozone to latest ratis snapshot build (0.4.0-5680cf5-SNAPSHOT). Contributed by Mukul Kumar Singh. 2019-03-14 15:18:06 +05:30
Ajay Yadav
4fa009989b
HDDS-1254. Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer. Contributed by Ajay Kumar. (#598) 2019-03-13 12:01:08 -07:00
Aravindan Vijayan
1f47fb7a2f HDDS-1209. Fix the block allocation logic in SCM when client wants to exclude all available open containers in a chosen pipeline. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-03-13 17:12:43 +05:30
Lokesh Jain
0d62753da9 HDDS-1128. Create stateful manager class for the pipeline creation scheduling. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-03-13 16:30:06 +05:30
Márton Elek
c449cdebe6
HDDS-807. Period should be an invalid character in bucket names. Contributed by Siddharth Wagle. 2019-03-12 19:14:55 +01:00
Ajay Yadav
dcb0de848d
HDDS-1043. Enable token based authentication for S3 api 
Closes #561
 2019-03-12 18:47:06 +01:00
Xiaoyu Yao
2778259b88 HDDS-1253. Fix checkstyle issue from Nightly run. Contributed by Xiaoyu Yao. (#593) 
Closes PR #593
 2019-03-12 09:34:18 -07:00
Mukul Kumar Singh
daf4660eeb HDDS-1095. OzoneManager#openKey should do multiple block allocations in a single SCM rpc call. Contributed by Mukul Kumar Singh. 2019-03-12 20:02:19 +05:30
Xiaoyu Yao
c7307867f0 HDDS-1245. OM delegation expiration time should use Time.now instead … (#587) 
Closes #587.
 2019-03-11 21:15:40 -07:00
Ajay Yadav
599e926d46 HDDS-596. Add robot test for OM Block Token. Contributed by Ajay Kumar. 
This closes (#581)
 2019-03-11 15:01:04 -07:00
Shashikant Banerjee
b4aa24d3c5 HDDS-1173. Fix a data corruption bug in BlockOutputStream. Contributed by Shashikant Banerjee. 2019-03-11 23:15:49 +05:30
Xiaoyu Yao
4fc493e511 HDDS-1236. Fix incorrect Ozone ClientProtocol KerberosInfo annotation. Contributed by Xiaoyu Yao. (#573) 2019-03-11 09:19:19 -07:00
Lokesh Jain
ebb5fa115b HDDS-1220. KeyManager#openKey should release the bucket lock before doing an allocateBlock. Contributed by Lokesh Jain. 2019-03-11 14:54:44 +05:30
Bharat Viswanadham
68aa2ccd89
HDDS-1240. Fix check style issues caused by HDDS-1196. 2019-03-09 21:50:46 -08:00
Bharat Viswanadham
73e1801b2d
HDDS-1242. In S3 when bucket already exists, it should just return location. 2019-03-09 21:42:17 -08:00
Elek, Márton
c072458e5d HDDS-1213. Support plain text S3 MPU initialization request. 2019-03-08 09:55:31 -08:00
Bharat Viswanadham
341c076f05 HDDS-1196. Add a ReplicationStartTimer class. (#567) 2019-03-08 20:55:33 +05:30
Yiqun Lin
fb851c9481 HDDS-1232. Recon Container DB service definition. Contributed by Aravindan Vijayan. 2019-03-08 16:59:41 +08:00
Vivek Ratnavel Subramanian
475011bbf8
HDDS-1226. ozone-filesystem jar missing in hadoop classpath 
Closes #560
 2019-03-07 14:09:13 +01:00
Hanisha Koneru
0eba4079bc
HDDS-1225. Provide docker-compose for OM HA. 
Closes #562
 2019-03-07 11:10:04 +01:00
Hanisha Koneru
bb12e81ec8
HDDS-1175. Serve read requests directly from RocksDB. (#557) 
HDDS-1175. Serve read requests directly from RocksDB.
 2019-03-06 19:44:55 -08:00
Vivek Ratnavel Subramanian
a55fc36299 HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values. 2019-03-06 17:43:57 -08:00
Márton Elek
45f976fdcf
HDDS-1113. Remove default dependencies from hadoop-ozone project. Contributed by Elek, Marton. 
Closes #490
 2019-03-06 18:22:22 +01:00
Mukul Kumar Singh
2c3ec37738 HDDS-1210. Ratis pipeline creation doesn't check raft client reply status during initialization. Contributed by Mukul Kumar Singh. 2019-03-06 21:38:21 +05:30
Ajay Kumar
9d87247af3
HDDS-1216. Change name of ozoneManager service in docker compose files to om. 
Closes #553
 2019-03-06 15:18:39 +01:00
Arpit Agarwal
1ad5bfc53f
HDDS-1188. Implement a skeleton patch for Recon server with initial set of interfaces. Contributed by Siddharth Wagle. 2019-03-05 15:54:29 -08:00
Elek, Márton
7f636b47b5 HDDS-919. Enable prometheus endpoints for Ozone datanodes (#502) 
Pull request from Elek, Márton. (#502)
 2019-03-05 12:04:57 -08:00
Anu Engineer
7fd890116a HDDS-1171. Add benchmark for OM and OM client in Genesis. 
Contributed by Lokesh Jain.
 2019-03-05 11:31:09 -08:00
Arpit Agarwal
3ef1235215
HDDS-1218. Do the dist-layout-stitching for Ozone after the test-compile phase. Contributed by Marton Elek. 2019-03-05 10:25:31 -08:00
Arpit Agarwal
549d54f9a3
Merge pull request #556 from elek/HDDS-1222 
HDDS-1222. Remove TestContainerSQLCli unit test stub
 2019-03-05 10:15:55 -08:00
Hanisha Koneru
12402b7a74 HDDS-1072. Implement RetryProxy and FailoverProxy for OM client. 2019-03-05 09:24:22 -08:00
Shashikant Banerjee
5fcea54a50 HDDS-935. Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart. Contributed by Shashikant Banerjee. 2019-03-05 22:09:46 +05:30
Ajay Yadav
353b90d59e HDDS-1156. testDelegationToken is failing in TestSecureOzoneCluster. Contributed by Ajay Kumar. 2019-03-05 08:37:10 -08:00
Márton Elek
02da06dd9f
HDDS-1222. Remove TestContainerSQLCli unit test stub. Contributed by Elek, Marton. 2019-03-05 16:35:04 +01:00
Anu Engineer
10b802b84b HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. 
Contributed by Aravindan Vijayan.
 2019-03-04 12:00:16 -08:00
Xiaoyu Yao
eed8b794d8 HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao. (#545) 2019-03-04 10:37:26 -08:00
Hanisha Koneru
b18c1c22ea
Revert "HDDS-1072. Implement RetryProxy and FailoverProxy for OM client." 
This reverts commit 8e1225991d.
 2019-03-04 16:58:59 +09:00
Ajay Yadav
9e53088493 Revert "HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contr…" (#544) 
* Revert "Fix checkstyle issue"

This reverts commit 78c8a3323e.

* Revert "HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao."

This reverts commit 6fef6fc5ee.
 2019-03-01 16:19:43 -08:00
Xiaoyu Yao
78c8a3323e Fix checkstyle issue 2019-03-01 16:15:20 -08:00
Xiaoyu Yao
6fef6fc5ee HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao. 2019-03-01 16:15:20 -08:00
Ajay Kumar
4a3cddea70 HDDS-134. SCM CA: OM sends CSR and uses certificate issued by SCM. Contributed by Ajay Kumar. 2019-03-01 15:41:09 -08:00
Shashikant Banerjee
de1dae64f2 HDDS-726. Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails. Contributed by Shashikant Banerjee. 2019-03-01 23:30:28 +05:30
Hanisha Koneru
8e1225991d HDDS-1072. Implement RetryProxy and FailoverProxy for OM client. 2019-03-01 00:27:39 -08:00
Anu Engineer
eae3db9649 HDDS-1191. Replace Ozone Rest client with S3 client in smoketests and docs. 
Contributed Elek, Marton.
 2019-02-28 17:14:47 -08:00
Xiaoyu Yao
3a8118b480
HDDS-1190. Fix jdk 11 issue for ozonesecure base image and docker-compose . Contributed by Xiaoyu Yao. 2019-02-28 15:51:08 +01:00
Ajay Kumar
cbf82fabf0 HDDS-1061. DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier. Contributed by Ajay Kumar. 2019-02-27 17:00:42 -08:00
Bharat Viswanadham
6c8c422beb
HDDS-1178. Healthy pipeline Chill Mode Rule. 
Closes #518
 2019-02-27 14:26:40 +01:00
Xiaoyu Yao
0e450202a6
HDDS-1019. Use apache/hadoop-runner image to test ozone secure cluster. Contributed by Xiaoyu Yao. 2019-02-27 12:44:04 +01:00
Ajay Kumar
625e93713b HDDS-1177. Add validation to AuthorizationHeaderV4. Contributed by Ajay Kumar 2019-02-26 18:57:43 -08:00
Anu Engineer
a5a751b418 HDDS-1174. Freon tests are failing with null pointer exception. 
Contributed by Shashikant Banerjee.
 2019-02-26 14:33:54 -08:00
Márton Elek
585cebf26b
HDDS-1104. Use picocli with Ozone genesis tool. Contributed by Lokesh Jain. 2019-02-26 16:11:55 +01:00
Xiaoyu Yao
9e0f3d1c52 HDDS-1038. Support Service Level Authorization for Ozone. Contributed by Xiaoyu Yao and Ajay Kumar. 2019-02-26 06:02:10 -08:00
Anu Engineer
9de34d2990 HDDS-1151. Propagate the tracing id in ScmBlockLocationProtocol. 
Contributed by Elek, Marton.
 2019-02-25 13:50:37 -08:00
Márton Elek
a5eb7ce651
HDDS-1170. Fix the FindBugs issue in OzoneClientAdapterFactory. Contributed by Anu Engineer. 2019-02-25 10:41:43 +01:00
Anu Engineer
92b1fdcece HDDS-1070. Adding Node and Pipeline related metrics in SCM. 
Contributed by Nandakumar.
 2019-02-24 15:39:41 -08:00
Anu Engineer
3aa0a57ea0 HDDS-1152. Add trace information for the client side of the datanode writes 
Contributed by Elek, Marton.
 2019-02-24 15:20:34 -08:00
Anu Engineer
5c1f946071 HDDS-1145. Add optional web server to the Ozone freon test tool. 
Contributed by Elek, Marton.
 2019-02-24 15:04:25 -08:00
Anu Engineer
236b105e41 HDDS-1089. Disable OzoneFSStorageStatistics for hadoop versions older than 2.8. 
Contributed by Elek, Marton.
 2019-02-24 13:53:38 -08:00
Anu Engineer
8387bbdfb5 HDDS-1154. Add tracing to the client side of StorageContainerLocationProtocol and OzoneManagerProtocol. 
Contributed by Elek, Marton.
 2019-02-24 13:13:37 -08:00
Bharat Viswanadham
106bdc6c04
HDDS-1120. Add a config to disable checksum verification during read. 2019-02-23 21:37:55 -08:00
Anu Engineer
c1e5b19212 HDDS-1147. Fix failing unit tests in TestOzoneManager. 
Contributed by Nandakumar.
 2019-02-23 20:56:24 -08:00
Anu Engineer
fb546d1807 HDDS-1167. Error in hadoop-ozone/dev-support/checks/checkstyle.sh. 
Contributed by Nandakumar.
 2019-02-23 20:45:01 -08:00
Hanisha Koneru
d0e21ac16b HDDS-1158. TestOzoneManagerHA.testTwoOMNodesDown is failing with ratis error. 2019-02-23 10:20:28 -08:00
Elek, Márton
28d0bf9f67 HDDS-1161. Disable failing test which are tracked by a separated jira. Contributed by Elek, Marton. 2019-02-21 13:57:05 -08:00
Bharat Viswanadham
d33f0666f6
HDDS-1141. Update DBCheckpointSnapshot to DBCheckpoint. 
* HDDS-1141.Update DBCheckpointSnapshot to DBCheckpoint.

* fix test failures in TestOzoneConfigurationFields
 2019-02-21 13:29:10 -08:00
Márton Elek
7c802c42dc
HDDS-1129. Fix findbug/checkstyle errors hdds projects. Contributed by Elek, Marton. 2019-02-21 14:04:18 +01:00
Hanisha Koneru
b1397ff9e4 HDDS-1109. Setup Failover Proxy Provider for OM client. 2019-02-20 14:49:59 -08:00
Arpit Agarwal
676a9cb888 HDDS-1053. Generate RaftGroupId from OMServiceID. Contributed by Aravindan Vijayan. 2019-02-20 12:57:49 -08:00
Xiaoyu Yao
1374f8f548 HDDS-1060. Add API to get OM certificate from SCM CA. Contributed by Ajay Kumar. 2019-02-20 11:35:49 -08:00
Márton Elek
642fe6a260
HDDS-1135. Ozone jars are missing in the Ozone Snapshot tar. Contributed by Dinesh Chitlangia. 2019-02-20 12:08:24 +01:00
Yiqun Lin
1d30fd94c6 HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by Lokesh Jain. 2019-02-20 10:45:51 +08:00
avijayanhwx
e8d7e3b4e6 HDDS-1139 : Fix findbugs issues caused by HDDS-1085. Contributed by Aravindan Vijayan. 2019-02-19 16:32:47 -08:00
Anu Engineer
02d04bd107 HDDS-1121. Key read failure when data is written parallel in to Ozone. 
Contributed by Bharat Viswanadham.
 2019-02-19 11:07:17 -08:00
Xiaoyu Yao
cf1a66d0da HDDS-1101. SCM CA: Write Certificate information to SCM Metadata. Contributed by Anu Engineer. 2019-02-19 10:42:56 -08:00
Anu Engineer
588b4c4d78 HDDS-1085. Create an OM API to serve snapshots to Recon server. 
Contributed by Aravindan Vijayan.
 2019-02-18 22:35:36 -08:00
Yiqun Lin
67af509097 HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure. Contributed by Yiqun Lin. 2019-02-19 11:29:52 +08:00
Yiqun Lin
f2fb6536dc HDDS-1106. Introduce queryMap in PipelineManager. Contributed by Lokesh Jain. 2019-02-18 22:35:23 +08:00
Anu Engineer
7ea91494ce HDDS-1041. Support TDE(Transparent Data Encryption) for Ozone. 
Contributed by Xiaoyu Yao.
 2019-02-16 10:23:10 -08:00
Elek, Márton
afe126d71f HDDS-1114. Fix findbugs/checkstyle/accepteance errors in Ozone. Contributed by Marton Elek. 2019-02-15 11:49:52 -08:00
Anu Engineer
217bdbd940 HDDS-1116.Add java profiler servlet to the Ozone web servers. 
Contributed by Elek, Marton.
 2019-02-15 11:32:06 -08:00
Elek, Márton
d10444e79c
HDDS-1092. Use Java 11 JRE to run Ozone in containers. 2019-02-15 17:26:56 +01:00
Shashikant Banerjee
de934ba2dc HDDS-1076. TestSCMNodeManager crashed the jvm. Contributed by Lokesh Jain. 2019-02-15 21:13:30 +05:30
Márton Elek
e0fe3d1eca HDDS-1110. OzoneManager need to login during init when security is enabled. Contributed by Xiaoyu Yao. 2019-02-15 16:12:24 +01:00
Anu Engineer
75e15cc0c4 HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects. 
Contributed by Elek, Marton.
 2019-02-14 23:33:25 -08:00
Anu Engineer
5cb67cf044 HDDS-1097. Add genesis benchmark for BlockManager#allocateBlock. 
Contributed by Lokesh Jain.
 2019-02-14 23:24:58 -08:00