big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,729 Commits 2 Branches 2 Tags 264 MiB
8c0759d02a
Commit Graph

441 Commits

Author SHA1 Message Date
Bharat Viswanadham
049bb42760 HDDS-1723. Create new OzoneManagerLock class. (#1006) 
* HDDS-1723. Create new OzoneManagerLock class.

* add logs

* fix exception message

* add license

* fix findbug

* throw exception when acquiring 2nd lock failed.

* fix review comments

* fix missed review comment

* fix checkstyle

* fix review comments
 2019-06-25 08:48:03 -07:00
Elek, Márton
004207996c HDDS-1597. Remove hdds-server-scm dependency from ozone-common (#969) 
* HDDS-1597. Remove hdds-server-scm dependency from ozone-common. Contributed by Elek, Marton.

* checkstyle fixes

* revert the import reorder of HddsUtil

* add javadoc

* switch back to the commons-lang2

* fix typo

* fix metrics core classpath problem (+rebase fix)
 2019-06-24 11:56:04 -07:00
avijayanhwx
81d2f1b724 HDDS-1719 : Increase ratis log segment size to 1MB. (#1005) 
* HDDS-1719 : Increase ratis log segment size to 1MB.

* HDDS-1719 : Increase ratis log segment size to 1MB.
 2019-06-24 08:57:08 -07:00
S O'Donnell
b95a58e231
HDDS-1674. Make ScmBlockLocationProtocol message type based. 
Closes #984
 2019-06-21 11:35:43 +02:00
avijayanhwx
71edfce187 HDDS-1699. Update RocksDB version to 6.0.1 (#980) 2019-06-18 11:13:13 -07:00
Bharat Viswanadham
ba681bb80e
HDDS-1692. RDBTable#iterator should disabled caching of the keys during iterator. (#975) 2019-06-16 22:00:39 -07:00
Bharat Viswanadham
88c53d516c
HDDS-1620. Implement Volume Write Requests to use Cache and DoubleBuffer. (#884) 2019-06-12 17:45:42 -07:00
Sammi Chen
1732312f45 HDDS-1663. Add datanode to network topology cluster during node regis… (#937) 2019-06-12 12:16:42 -07:00
Márton Elek
f918e3fe62
HDDS-1669. SCM startup is failing if network-topology-default.xml is part of a jar 
Closes #946
 2019-06-11 19:30:22 +02:00
Elek, Márton
46b23c11b0 HDDS-1636. Tracing id is not propagated via async datanode grpc call (#895) 2019-06-07 20:40:32 -07:00
Hanisha Koneru
a91d24fea4
HDDS-1496. Support partial chunk reads and checksum verification (#804) 2019-06-06 19:44:40 -07:00
Hanisha Koneru
0b115b60b0 HDDS-1371. OMSnapshotProvider to download DB checkpoint from leader OM. (#703) 2019-06-06 19:42:34 -07:00
Sammi Chen
8ca58efeec HDDS-1490. Support configurable container placement policy through 'o… (#903) 2019-06-06 12:14:47 -07:00
avijayanhwx
fe069570d8 HDDS-1647 : Recon config tag does not show up on Ozone UI. (#914) 2019-06-06 23:43:29 +05:30
Bharat Viswanadham
0b8a2608e0 HDDS-1605. Implement AuditLogging for OM HA Bucket write requests. (#867) 2019-06-06 10:13:36 -07:00
Sammi Chen
73954c1dd9 HDDS-1612. Add 'scmcli printTopology' shell command to print datanode topology. Contributed by Sammi Chen.(#910) 2019-06-05 20:13:39 -07:00
avijayanhwx
1a78794227 HDDS-1624 : Refactor operations inside the bucket lock in OM key write. (#882) 2019-06-04 15:41:37 -07:00
bshashikant
e4b4fec66f HDDS-1224. Restructure code to validate the response from server in the Read path (#806) 2019-06-04 10:37:02 -07:00
Márton Elek
2a97a37d9e
Revert "HDDS-1597. Remove hdds-server-scm dependency from ozone-common. (#860)" 
This reverts commit baee71551d.
 2019-06-03 08:34:19 +02:00
Márton Elek
1ae062c818
HDDS-1382. Create customized CSI server for Ozone 
Closes #693
 2019-05-31 13:35:58 +02:00
Bharat Viswanadham
c965f7f499
HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. (#850) 2019-05-30 15:28:15 -07:00
Elek, Márton
baee71551d HDDS-1597. Remove hdds-server-scm dependency from ozone-common. (#860) 2019-05-30 09:20:16 -07:00
avijayanhwx
33419a980a HDDS-1568 : Add RocksDB metrics to OM. Contributed by Aravindan Vijayan 2019-05-30 08:48:14 -07:00
dineshchitlangia
0b907bc0e8 HDDS-1581. Atleast one of the metadata dir config property must be tagged as REQUIRED. 2019-05-30 08:39:19 -07:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
avijayanhwx
4b099b8b89 HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM. (#819) 2019-05-23 15:34:55 -07:00
Xiaoyu Yao
20a4ec351c HDDS-700. Support rack awared node placement policy based on network topology. Contributed by Sammi Chen. 2019-05-23 10:40:15 -07:00
Ajay Yadav
9c61494c02 HDDS-1065. OM and DN should persist SCM certificate as the trust root. Contributed by Ajay Kumar. (#834) 2019-05-22 11:47:32 -07:00
Ajay Yadav
ef1cc725b8
HDDS-1538. Update ozone protobuf message for ACLs. Contributed by Ajay Kumar. (#828) 2019-05-21 15:53:40 -07:00
Lokesh Jain
c1d7d68c78
HDDS-1461. Optimize listStatus api in OzoneFileSystem (#782) 2019-05-21 14:47:48 +05:30
Bharat Viswanadham
0d1d7c86ec HDDS-1499. OzoneManager Cache. (#798) 2019-05-19 19:23:02 -07:00
Márton Elek
9248b7d58b
HDDS-1284. Adjust default values of pipline recovery for more resilient service restart. 2019-05-16 17:04:46 +02:00
Shashikant Banerjee
e66ecc564a HDDS-1531. Disable the sync flag by default during chunk writes in Datanode (#820). Contributed by Shashikant Banerjee. 2019-05-16 18:38:52 +05:30
Vivek Ratnavel Subramanian
b832e174b0 HDDS-1474. "ozone.scm.datanode.id" config should take path for a dir and not a file (#809) 2019-05-09 17:11:09 -07:00
Hanisha Koneru
2d31ccc1b3
Revert "HDDS-1474. ozone.scm.datanode.id config should take path for a dir (#792)" (#808) 
This reverts commit 18cca583ac.
 2019-05-09 11:18:46 -07:00
Vivek Ratnavel Subramanian
18cca583ac HDDS-1474. ozone.scm.datanode.id config should take path for a dir (#792) 2019-05-09 10:23:44 -07:00
Shashikant Banerjee
7f0e2c67e0 HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749). Contributed by Shashikant Banerjee 2019-05-08 01:00:10 +05:30
Shashikant Banerjee
01451a57de Revert "HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749)" 
This reverts commit d525633619.
 2019-05-08 00:49:15 +05:30
bshashikant
d525633619
HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749) 
HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749).
 2019-05-08 00:38:53 +05:30
Márton Elek
e2f0f72677
HDDS-1469. Generate default configuration fragments based on annotations 
Closes #773
 2019-05-02 12:14:43 +02:00
Márton Elek
a2887f5c23
HDDS-1468. Inject configuration values to Java objects 
Closes #772
 2019-05-02 11:35:01 +02:00
Xiaoyu Yao
a96d16a843
HDDS-1455. Inconsistent naming convention with Ozone Kerberos configuration. Contributed by Xiaoyu Yao. (#762) 2019-04-29 11:18:11 -07:00
Hanisha Koneru
37582705fa
HDDS-1403. KeyOutputStream writes fails after max retries while writing to a closed container (#753) 2019-04-26 10:39:03 -07:00
Kitti Nanasi
3f787cd506
HDDS-1192. Support -conf command line argument in GenericCli 
Closes #713
 2019-04-24 15:34:49 +02:00
Xiaoyu Yao
64f30da428 HDDS-1450. Fix nightly run failures after HDDS-976. Contributed by Xiaoyu Yao. (#757) 2019-04-23 21:38:23 -07:00
Chen, Junjie
518f47bf9b HDDS-976: Parse network topology from yaml file. Contributed by Junjie Chen. (#661) 
Signed-off-by: Xiaoyu Yao <xyao@apache.org>
 2019-04-18 16:53:25 -07:00
Márton Elek
8aea5e4cb6
HDDS-1433. Rename GetScmInfoRespsonseProto to GetScmInfoResponseProto due to typos. Contributed by Wanqiang Ji. 2019-04-18 08:53:08 +02:00
Shashikant Banerjee
d608be660f HDDS-1373. KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException.(#729) 2019-04-17 14:49:02 +05:30
Shashikant Banerjee
082f1e0437 Revert "HDDS-1373. KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException..(#729)" 
This reverts commit df2ae27f3e.
 2019-04-17 14:38:08 +05:30
Shashikant Banerjee
df2ae27f3e HDDS-1373. KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException..(#729) 2019-04-17 14:25:35 +05:30
Shashikant Banerjee
e5a918cda3 Revert "Merge pull request #729 from bshashikant/HDDS-1373" 
This reverts commit 4ff4314e40, reversing
changes made to 04c0437d13.
 2019-04-17 14:17:54 +05:30
bshashikant
4ff4314e40
Merge pull request #729 from bshashikant/HDDS-1373 
HDDS-1373. KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException. Contributed by Shashikant Banerjee.
 2019-04-17 14:12:52 +05:30
Arpit Agarwal
ebbda181e4
HDDS-1198. Rename chill mode to safe mode. Contributed by Siddharth Wagle. (#737) 2019-04-13 07:27:03 -07:00
Arpit Agarwal
a9b46c58fe
HDDS-1421. Avoid unnecessary object allocations in TracingUtil 
Closes #722
 2019-04-12 11:52:31 +02:00
Doroszlai, Attila
3b08ac4659 HDDS-1363. ozone.metadata.dirs doesn't pick multiple dirs (#691) 2019-04-12 15:00:08 +05:30
Nanda kumar
c298415eb5
HDDS-1404. Fix typos in HDDS. Contributed by bianqi. 2019-04-12 14:12:10 +05:30
Shashikant Banerjee
22d0468b39 HDDS-1373. KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException. Contributed by Shashikant Banerjee 2019-04-11 22:13:41 +05:30
Shashikant Banerjee
2d4f6b6daa HDDS-372. There are three buffer copies in BlockOutputStream. Contributed by Shashikant Banerjee. 2019-04-09 10:27:58 +05:30
Nanda kumar
48a58bce37
HDDS-1207. Refactor Container Report Processing logic and plugin new Replication Manager. (#662) 2019-04-04 16:32:59 +05:30
Yiqun Lin
a92806d05a HDDS-1189. Recon Aggregate DB schema and ORM. Contributed by Siddharth Wagle. 2019-04-04 17:33:37 +08:00
Hanisha Koneru
f09a78f73f
HDDS-1339. Implement ratis snapshots on OM (#651) 2019-04-03 22:50:28 -07:00
Kitti Nanasi
aeb43dfba3
HDDS-1153. Make tracing instrumentation configurable 
Closes #673
 2019-04-02 14:21:47 +02:00
Shashikant Banerjee
ac4010bb22 HDDS-1293. ExcludeList#getProtoBuf throws ArrayIndexOutOfBoundsException. Contributed by Shashikant Banerjee. 2019-03-29 00:43:28 +05:30
Xiaoyu Yao
ca5e4ce036
HDDS-1318. Fix MalformedTracerStateStringException on DN logs. Contributed by Xiaoyu Yao. 
This closes #641
 2019-03-28 12:00:58 -07:00
Yiqun Lin
e5d72f504e HDDS-1234. Iterate the OM DB snapshot and populate the recon container DB. Contributed by Aravindan Vijayan. 2019-03-25 22:52:02 +08:00
Shashikant Banerjee
d4e4a7d456 HDDS-1317. KeyOutputStream#write throws ArrayIndexOutOfBoundsException when running RandomWrite MR examples. Contributed by Shashikant Banerjee. 2019-03-25 15:41:20 +05:30
Arpit Agarwal
f854a89190
HDDS-1205. Refactor ReplicationManager to handle QUASI_CLOSED containers. Contributed by Nanda kumar. (#620) 2019-03-22 14:36:29 -07:00
Yiqun Lin
60cdd4cac1 HDDS-1233. Create an Ozone Manager Service provider for Recon. Contributed by Aravindan Vijayan. 2019-03-21 11:02:29 +08:00
Ajay Yadav
f10d493325
HDDS-1119. DN get OM certificate from SCM CA for block token validation. Contributed by Ajay Kumar. (#601) 2019-03-18 23:08:17 -07:00
Arpit Agarwal
ae3a2c3851
Revert "HDDS-1284. Adjust default values of pipline recovery for more resilient service restart. Contributed by Elek, Marton. (#608)" 
This reverts commit 44b8451821.
 2019-03-18 14:21:57 -07:00
Sammi Chen
4d2a116d6e HDDS-699. Detect Ozone Network topology. Contributed by Sammi Chen. 2019-03-17 17:08:29 +08:00
Elek, Márton
44b8451821 HDDS-1284. Adjust default values of pipline recovery for more resilient service restart. Contributed by Elek, Marton. (#608) 2019-03-15 14:51:37 -07:00
Shashikant Banerjee
155ab6d5d8 HDDS-1098. Introduce Retry Policy in Ozone Client. Contributed by Shashikant Banerjee. 2019-03-15 15:40:59 +05:30
Márton Elek
3d140a537a
HDDS-1247. Bump trunk ozone version to 0.5.0. Contributed by Elek, Marton. 
Closes #588
 2019-03-14 12:01:25 +01:00
Xiaoyu Yao
86d508c7c7 HDDS-1087. Fix TestDefaultCertificateClient#testSignDataStream. Contributed by Xiaoyu Yao. (#596) 2019-03-13 11:50:53 -07:00
Vivek Ratnavel Subramanian
c1649bf1e1
HDDS-1226. Addendum. ozone-filesystem jar missing in hadoop classpath 
Closes #560
 2019-03-13 12:08:45 +01:00
Lokesh Jain
0d62753da9 HDDS-1128. Create stateful manager class for the pipeline creation scheduling. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-03-13 16:30:06 +05:30
Arpit Agarwal
24793d2d97
HDDS-1163. Basic framework for Ozone Data Scrubber. Contributed by Supratim Deka. 2019-03-13 04:32:39 +09:00
Ajay Yadav
dcb0de848d
HDDS-1043. Enable token based authentication for S3 api 
Closes #561
 2019-03-12 18:47:06 +01:00
Xiaoyu Yao
2778259b88 HDDS-1253. Fix checkstyle issue from Nightly run. Contributed by Xiaoyu Yao. (#593) 
Closes PR #593
 2019-03-12 09:34:18 -07:00
Mukul Kumar Singh
daf4660eeb HDDS-1095. OzoneManager#openKey should do multiple block allocations in a single SCM rpc call. Contributed by Mukul Kumar Singh. 2019-03-12 20:02:19 +05:30
Márton Elek
d17e31e062
HDDS-1214. Enable tracing for the datanode read/write path. Contributed by Elek, Marton. 
Closes #550.
 2019-03-12 11:34:56 +01:00
Ajay Yadav
599e926d46 HDDS-596. Add robot test for OM Block Token. Contributed by Ajay Kumar. 
This closes (#581)
 2019-03-11 15:01:04 -07:00
Bharat Viswanadham
68aa2ccd89
HDDS-1240. Fix check style issues caused by HDDS-1196. 2019-03-09 21:50:46 -08:00
Bharat Viswanadham
341c076f05 HDDS-1196. Add a ReplicationStartTimer class. (#567) 2019-03-08 20:55:33 +05:30
Yiqun Lin
fb851c9481 HDDS-1232. Recon Container DB service definition. Contributed by Aravindan Vijayan. 2019-03-08 16:59:41 +08:00
Ajay Yadav
064f38b3a5 HDDS-594. SCM CA: DN sends CSR and uses certificate issued by SCM. Contributed by Ajay Kumar. (#547) 2019-03-07 14:41:52 -08:00
Hanisha Koneru
bb12e81ec8
HDDS-1175. Serve read requests directly from RocksDB. (#557) 
HDDS-1175. Serve read requests directly from RocksDB.
 2019-03-06 19:44:55 -08:00
Vivek Ratnavel Subramanian
a55fc36299 HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values. 2019-03-06 17:43:57 -08:00
Shashikant Banerjee
62e89dc275 HDDS-1184. Parallelization of write chunks in datanodes is broken. Contributed by Shashikant Banerjee. 2019-03-06 10:00:16 +05:30
Arpit Agarwal
1ad5bfc53f
HDDS-1188. Implement a skeleton patch for Recon server with initial set of interfaces. Contributed by Siddharth Wagle. 2019-03-05 15:54:29 -08:00
Elek, Márton
7f636b47b5 HDDS-919. Enable prometheus endpoints for Ozone datanodes (#502) 
Pull request from Elek, Márton. (#502)
 2019-03-05 12:04:57 -08:00
Hanisha Koneru
12402b7a74 HDDS-1072. Implement RetryProxy and FailoverProxy for OM client. 2019-03-05 09:24:22 -08:00
Shashikant Banerjee
5fcea54a50 HDDS-935. Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart. Contributed by Shashikant Banerjee. 2019-03-05 22:09:46 +05:30
Anu Engineer
10b802b84b HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. 
Contributed by Aravindan Vijayan.
 2019-03-04 12:00:16 -08:00
Xiaoyu Yao
eed8b794d8 HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao. (#545) 2019-03-04 10:37:26 -08:00
Hanisha Koneru
b18c1c22ea
Revert "HDDS-1072. Implement RetryProxy and FailoverProxy for OM client." 
This reverts commit 8e1225991d.
 2019-03-04 16:58:59 +09:00
Ajay Yadav
9e53088493 Revert "HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contr…" (#544) 
* Revert "Fix checkstyle issue"

This reverts commit 78c8a3323e.

* Revert "HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao."

This reverts commit 6fef6fc5ee.
 2019-03-01 16:19:43 -08:00
Xiaoyu Yao
6fef6fc5ee HDDS-1183. Override getDelegationToken API for OzoneFileSystem. Contributed by Xiaoyu Yao. 2019-03-01 16:15:20 -08:00