Commit Graph

887 Commits

Author SHA1 Message Date
Anu Engineer
4b50981cfa HDDS-1709. TestScmSafeNode is flaky.
Contributed by Elek, Marton.
2019-06-25 13:57:01 -07:00
Bharat Viswanadham
049bb42760 HDDS-1723. Create new OzoneManagerLock class. (#1006)
* HDDS-1723. Create new OzoneManagerLock class.

* add logs

* fix exception message

* add license

* fix findbug

* throw exception when acquiring 2nd lock failed.

* fix review comments

* fix missed review comment

* fix checkstyle

* fix review comments
2019-06-25 08:48:03 -07:00
Elek, Márton
d023f1f864 HDDS-1646. Support real persistence in the k8s example files (#945)
* HDDS-1646. Support real persistence in the k8s example files

* ephemeral clusters can be scaled up
2019-06-24 12:02:35 -07:00
Elek, Márton
004207996c HDDS-1597. Remove hdds-server-scm dependency from ozone-common (#969)
* HDDS-1597. Remove hdds-server-scm dependency from ozone-common. Contributed by Elek, Marton.

* checkstyle fixes

* revert the import reorder of HddsUtil

* add javadoc

* switch back to the commons-lang2

* fix typo

* fix metrics core classpath problem (+rebase fix)
2019-06-24 11:56:04 -07:00
S O'Donnell
cba13c7f83
HDDS-1715. Update the Intellij runner definitition of SCM to use the new class name
Closes #1000
2019-06-21 16:25:10 +02:00
Márton Elek
256bdd6982
HDDS-1678. Default image name for kubernetes examples should be ozone and not hadoop
Closes #959
2019-06-21 14:14:06 +02:00
Bharat Viswanadham
d9a9e9913e
HDDS-1579. Create OMDoubleBuffer metrics. (#871) 2019-06-20 09:42:45 -07:00
Márton Elek
9c4b15d2f4
HDDS-1508. Provide example k8s deployment files for the new CSI server
Closes #905
2019-06-20 16:36:42 +02:00
Hanisha Koneru
2dfa932818
HDDS-1684. OM should create Ratis related dirs only if ratis is enabled (#965) 2019-06-18 16:08:48 -07:00
Vivek Ratnavel Subramanian
37bd5bb342 HDDS-1702. Optimize Ozone Recon build time (#982) 2019-06-18 14:44:23 -07:00
Vivek Ratnavel Subramanian
fb1ce0d50a HDDS-1670. Add limit support to /api/containers and /api/containers/{id} endpoints (#954) 2019-06-18 08:51:16 -07:00
Xudong Cao
3d020e914f HDDS-1532. Improve the concurrent testing framework of Freon. (#957) 2019-06-17 11:09:05 -07:00
S O'Donnell
22b36dd311
HDDS-1660. Use Picocli for Ozone Manager
Closes #925
2019-06-17 14:13:50 +02:00
Bharat Viswanadham
e70aeb4d7e
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. (#972) 2019-06-15 13:47:07 -07:00
Elek, Márton
89acbbc0b9 HDDS-1677. Auditparser robot test shold use a world writable working directory (#958) 2019-06-13 16:04:14 -07:00
Chen, Junjie
940bcf01cd HDDS-1587. Support dynamically adding delegated classes from to isolated class loader. Contributed by Junjie Chen. (#942) 2019-06-12 20:08:15 -07:00
Bharat Viswanadham
88c53d516c
HDDS-1620. Implement Volume Write Requests to use Cache and DoubleBuffer. (#884) 2019-06-12 17:45:42 -07:00
Xiaoyu Yao
a43f4440f7
HDDS-1543. Implement addAcl,removeAcl,setAcl,getAcl for Prefix. Contr… (#927) 2019-06-12 16:25:31 -07:00
Sammi Chen
1732312f45 HDDS-1663. Add datanode to network topology cluster during node regis… (#937) 2019-06-12 12:16:42 -07:00
Ajay Yadav
3b31694c35 HDDS-1545. Cli to add,remove,get and delete acls for Ozone objects. Contributed by Ajay Kumar. (#920) 2019-06-12 06:51:34 -07:00
Elek, Márton
96e1e41746 HDDS-1627. Make the version of the used hadoop-runner configurable (#886) 2019-06-11 11:18:12 -07:00
Elek, Márton
3c9a5e7b16 HDDS-1635. Maintain docker entrypoint and envtoconf inside ozone project (#894) 2019-06-11 11:12:05 -07:00
Márton Elek
f918e3fe62
HDDS-1669. SCM startup is failing if network-topology-default.xml is part of a jar
Closes #946
2019-06-11 19:30:22 +02:00
Sammi Chen
b05747901f
HDDS-1662. Missing test resources of integrataion-test project in target directory after compile
Closes #933
2019-06-11 12:25:42 +02:00
Ajay Yadav
9deac3b6bf HDDS-1657. Fix parallelStream usage in volume and key native acl. Contributed by Ajay Kumar. (#926) 2019-06-07 21:47:14 -07:00
Elek, Márton
46b23c11b0 HDDS-1636. Tracing id is not propagated via async datanode grpc call (#895) 2019-06-07 20:40:32 -07:00
S O'Donnell
85479577da
HDDS-1622. Use picocli for StorageContainerManager
Closes #918
2019-06-07 17:50:57 +02:00
Hanisha Koneru
a91d24fea4
HDDS-1496. Support partial chunk reads and checksum verification (#804) 2019-06-06 19:44:40 -07:00
Hanisha Koneru
0b115b60b0 HDDS-1371. OMSnapshotProvider to download DB checkpoint from leader OM. (#703) 2019-06-06 19:42:34 -07:00
Sammi Chen
8ca58efeec HDDS-1490. Support configurable container placement policy through 'o… (#903) 2019-06-06 12:14:47 -07:00
Xiaoyu Yao
76c0183ae3 HDDS-1652. HddsDispatcher should not shutdown volumeSet. Contributed by Xiaoyu Yao. (#916) 2019-06-06 23:47:59 +05:30
Bharat Viswanadham
0b8a2608e0 HDDS-1605. Implement AuditLogging for OM HA Bucket write requests. (#867) 2019-06-06 10:13:36 -07:00
S O'Donnell
36757ad6a9
HDDS-1645. Change the version of Pico CLI to the latest 3.x release - 3.9.6
Closes #909
2019-06-06 16:00:00 +02:00
Márton Elek
f7c77b395f
HDDS-1458. Create a maven profile to run fault injection tests. Contributed by Eric Yang. 2019-06-06 12:24:13 +02:00
Ajay Yadav
3b1c2577d7 HDDS-1541. Implement addAcl,removeAcl,setAcl,getAcl for Key. Contributed by Ajay Kumat. (#885) 2019-06-05 14:42:10 -07:00
Márton Elek
42cd861be0
HDDS-1628. Fix the execution and return code of smoketest executor shell script
Closes #902
2019-06-05 14:04:17 +02:00
Vivek Ratnavel Subramanian
ef2da40a70
HDDS-1640. Reduce the size of recon jar file
Closes #907
2019-06-05 07:51:23 +02:00
avijayanhwx
1a78794227 HDDS-1624 : Refactor operations inside the bucket lock in OM key write. (#882) 2019-06-04 15:41:37 -07:00
Bharat Viswanadham
580b639908
HDDS-1600. Add userName and IPAddress as part of OMRequest. (#857) 2019-06-04 12:31:44 -07:00
Elek, Márton
97607f3dce HDDS-1510. Classpath files are deployed to the maven repository as pom/jar files. Contributed by Elek, Marton. (#900) 2019-06-04 10:38:37 -07:00
bshashikant
e4b4fec66f HDDS-1224. Restructure code to validate the response from server in the Read path (#806) 2019-06-04 10:37:02 -07:00
Elek, Márton
50909a7aa0 HDDS-1641. Csi server fails because transitive Netty dependencies (#906) 2019-06-04 09:18:15 -05:00
Elek, Márton
5d5081eff8 HDDS-1631. Fix auditparser smoketests (#892) 2019-06-04 01:30:43 -05:00
Elek, Márton
e140a45046 HDDS-1629. Tar file creation can be optional for non-dist builds. Contributed by Elek, Marton. (#887) 2019-06-04 01:20:45 -05:00
Elek, Márton
1fc359fc10 HDDS-1607. Create smoketest for non-secure mapreduce example (#869)
* HDDS-1607. Create smoketest for non-secure mapreduce example.

* remove hardcoded project version
2019-06-04 01:18:02 -05:00
avijayanhwx
21de9af903 HDDS-1625 : ConcurrentModificationException when SCM has containers of different owners. (#883) 2019-06-04 01:15:04 +05:30
Shashikant Banerjee
f3271126fc HDDS-1558. IllegalArgumentException while processing container Reports.
Signed-off-by: Nanda kumar <nanda@apache.org>
2019-06-04 00:59:02 +05:30
Márton Elek
2a97a37d9e
Revert "HDDS-1597. Remove hdds-server-scm dependency from ozone-common. (#860)"
This reverts commit baee71551d.
2019-06-03 08:34:19 +02:00
Ajay Yadav
c1d2d92187 HDDS-1540. Implement addAcl,removeAcl,setAcl,getAcl for Bucket. Contributed by Ajay Kumar. (#874) 2019-05-31 14:08:28 -07:00
Márton Elek
1ae062c818
HDDS-1382. Create customized CSI server for Ozone
Closes #693
2019-05-31 13:35:58 +02:00
Xiaoyu Yao
219e286722
HDDS-1608. Support Ozone Prefix ACLs in OM metadata table. Contributed by Xiaoyu Yao. (#875) 2019-05-30 16:44:38 -07:00
Bharat Viswanadham
c965f7f499
HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. (#850) 2019-05-30 15:28:15 -07:00
Elek, Márton
baee71551d HDDS-1597. Remove hdds-server-scm dependency from ozone-common. (#860) 2019-05-30 09:20:16 -07:00
Márton Elek
1b041d4fd4
HDDS-1497. Refactor blockade Tests. Contributed by Nilotpal Nandi. 2019-05-30 16:46:06 +02:00
Shashikant Banerjee
8c8cb2d6aa HDDS-1502. Fix the compilation issue with commit 18c1eebc08. 2019-05-30 18:44:22 +05:30
Shashikant Banerjee
18c1eebc08 HDDS-1502. Add metrics for Ozone Ratis performance.Contributed by Shashikant Banerjee(#833). 2019-05-30 16:17:45 +05:30
Ajay Yadav
12be6ff2ff HDDS-1539. Implement addAcl,removeAcl,setAcl,getAcl for Volume. Contributed Ajay Kumar. (#847) 2019-05-29 23:18:06 -07:00
Xudong Cao
9ad7cad205 HDDS-1530. Freon support big files larger than 2GB and add --bufferSize and --validateWrites options. Contributed by Xudong Cao. (#830) 2019-05-29 15:57:48 -07:00
Xiaoyu Yao
0ead2090a6 HDDS-1542. Create Radix tree to support ozone prefix ACLs. Contributed by Xiaoyu Yao. 2019-05-29 22:39:27 +02:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
Xiaoyu Yao
7f2e87a419
HDDS-1604. ContainerReader#initializeUsedBytes leaks DB reference. Co… (#866) 2019-05-28 18:39:23 -07:00
Elek, Márton
79d14d0d42 HDDS-1341. TestContainerReplication#testContainerReplication fails intermittently. Contributed by Elek, Marton. (#862) 2019-05-28 14:40:45 -07:00
Xiaoyu Yao
fb0b39f4bf
HDDS-1536. testSCMSafeModeRestrictedOp is failing consistently. Contributed by Xiaoyu Yao. (#865) 2019-05-28 14:02:36 -07:00
Márton Elek
06fa4d2764
HDDS-1565. Rename k8s-dev and k8s-dev-push profiles to docker and docker-push.
Closes: #840
2019-05-28 16:53:54 +02:00
Mukul Kumar Singh
72dd79015a HDDS-1534. freon should return non-zero exit code on failure. Contributed by Nilotpal Nandi. 2019-05-28 07:52:56 +05:30
Shashikant Banerjee
83549dbbea HDDS-1509. TestBlockOutputStreamWithFailures#test2DatanodesFailure fails intermittently. Contributed by Shashikant Banerjee (#805). 2019-05-27 16:31:44 +05:30
Shashikant Banerjee
f0e44b3a3f HDDS-1584. Fix TestFailureHandlingByClient tests. Contributed by Shashikant Banerjee (#845). 2019-05-27 16:26:21 +05:30
Eric Yang
a3745c59a3
HDDS-1588. Update Ozone dist image to use jdk11. Contributed by Eric Yang. 2019-05-27 10:18:42 +02:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Bharat Viswanadham
6d0e79c121
HDDS-1512. Implement DoubleBuffer in OzoneManager. (#810) 2019-05-24 10:50:54 -07:00
avijayanhwx
4b099b8b89 HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM. (#819) 2019-05-23 15:34:55 -07:00
Ajay Yadav
9c61494c02 HDDS-1065. OM and DN should persist SCM certificate as the trust root. Contributed by Ajay Kumar. (#834) 2019-05-22 11:47:32 -07:00
Mukul Kumar Singh
2fc6f8599a HDDS-1449. JVM Exit in datanode while committing a key. Contributed by Mukul Kumar Singh. (#825) 2019-05-22 17:18:40 +05:30
Bharat Viswanadham
77c49f2963
HDDS-1406. Avoid usage of commonPool in RatisPipelineUtils. (#714) 2019-05-21 18:41:30 -07:00
Ajay Yadav
ef1cc725b8
HDDS-1538. Update ozone protobuf message for ACLs. Contributed by Ajay Kumar. (#828) 2019-05-21 15:53:40 -07:00
Ajay Kumar
ff5691ca06
HDDS-1442. add spark container to ozonesecure-mr compose files
Closes #746
2019-05-21 17:10:46 +02:00
Lokesh Jain
c1d7d68c78
HDDS-1461. Optimize listStatus api in OzoneFileSystem (#782) 2019-05-21 14:47:48 +05:30
Vivek Ratnavel Subramanian
05db2a598e HDDS-1487. Bootstrap React framework for Recon UI (#831)
HDDS-1487. Bootstrap React framework for Recon UI.
Contributed by Vivek Ratnavel Subramanian.
2019-05-20 22:09:58 +02:00
Bharat Viswanadham
0d1d7c86ec HDDS-1499. OzoneManager Cache. (#798) 2019-05-19 19:23:02 -07:00
Siddharth
c183bd8e20 HDDS-1527. HDDS Datanode start fails due to datanode.id file read error. Contributed by Siddharth Wagle.
This closes #822.
2019-05-16 15:13:10 -07:00
Elek, Márton
b7de735159 HDDS-1522. Provide intellij runConfiguration for Ozone components. Contributed by Elek, Marton. 2019-05-16 16:38:50 +02:00
supratimdeka
9569015802 HDDS-1511. Space tracking for Open Containers in HDDS Volumes. Contributed by Supratim Deka (#812) 2019-05-15 10:26:01 -07:00
Lokesh Jain
6bcc1dce52
HDDS-1503. Reduce garbage generated by non-netty threads in datanode ratis server (#813) 2019-05-14 14:49:50 +05:30
Hanisha Koneru
02c9efcb81
HDDS-1491. Ozone KeyInputStream seek() should not read the chunk file. (#795) 2019-05-13 20:49:52 -07:00
Vivek Ratnavel Subramanian
b832e174b0 HDDS-1474. "ozone.scm.datanode.id" config should take path for a dir and not a file (#809) 2019-05-09 17:11:09 -07:00
Hanisha Koneru
2d31ccc1b3
Revert "HDDS-1474. ozone.scm.datanode.id config should take path for a dir (#792)" (#808)
This reverts commit 18cca583ac.
2019-05-09 11:18:46 -07:00
Vivek Ratnavel Subramanian
18cca583ac HDDS-1474. ozone.scm.datanode.id config should take path for a dir (#792) 2019-05-09 10:23:44 -07:00
avijayanhwx
25951255ce HDDS-1500. Allocate block failures in client should print exception trace. Contributed by Aravindan Vijayan (#801). 2019-05-09 15:38:49 +05:30
Siddharth
c4be3ea276 HDDS-1489. Unnecessary log messages on console with Ozone shell. (#797) 2019-05-07 14:48:39 -07:00
avijayanhwx
eb9c8900bc HDDS-1475 : Fix OzoneContainer start method. (#788) 2019-05-07 14:09:41 -07:00
Shashikant Banerjee
7f0e2c67e0 HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749). Contributed by Shashikant Banerjee 2019-05-08 01:00:10 +05:30
Shashikant Banerjee
01451a57de Revert "HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749)"
This reverts commit d525633619.
2019-05-08 00:49:15 +05:30
bshashikant
d525633619
HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749)
HDDS-1395. Key write fails with BlockOutputStream has been closed exception (#749).
2019-05-08 00:38:53 +05:30
Márton Elek
1a696cc761
HDDS-1424. Support multi-container robot test execution
Closes #726
2019-05-07 17:48:10 +02:00
Márton Elek
fb7c1cad0e
Revert "HDDS-1384. TestBlockOutputStreamWithFailures is failing"
This reverts commit dead9b4049.
2019-05-06 20:17:00 +02:00
Siddharth
1d70c8ca0f HDDS-1464. Client should have different retry policies for different exceptions. (#785) 2019-05-05 09:21:15 -07:00
avijayanhwx
f194540520 HDDS-1448 : RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation. (#786) 2019-05-04 00:19:00 +05:30
Siddharth
1df679985b HDDS-1473. DataNode ID file should be human readable. (#781) 2019-05-02 13:59:14 -07:00
Bharat Viswanadham
4605db369e
HDDS-1482. Use strongly typed codec implementations for the S3Table
Closes #789
2019-05-02 13:56:30 +02:00
Márton Elek
e2f0f72677
HDDS-1469. Generate default configuration fragments based on annotations
Closes #773
2019-05-02 12:14:43 +02:00