big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,944 Commits 2 Branches 2 Tags 264 MiB
f16cf877e5
Commit Graph

900 Commits

Author SHA1 Message Date
Elek, Márton
c7d426daf0 HDDS-1881. Design doc: decommissioning in Ozone (#1196) 
* Address comments from Anu
 * In-flight calculation is removed from the base model
 * Throttling is mentioned but will be discussed in a separated document
 * Model is improved to handle the over-replication in a specific way
 (to handle Nanda's use-case)
 * Stop conditions are defined independently from the model
 * Style of the table fragments are improved
* typo fix (thx to sodonnell)
* remove white spaces from the end of the lines
* example is restored
* minor whitespace improvement to support pdf conversion
 2019-08-28 10:19:03 -07:00
supratimdeka
1407414a52 HDDS-1094. Performance test infrastructure : skip writing user data on Datanode. Contributed by Supratim Deka (#1323) 2019-08-28 10:05:20 -07:00
Anu Engineer
dc72782008 HDDS-738. Removing REST protocol support from OzoneClient. 
Contributed by Elek, Marton.
 2019-08-28 09:53:31 -07:00
Márton Elek
c0499bd704 HDDS-1596. Create service endpoint to download configuration from SCM. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-28 09:13:46 -07:00
bshashikant
5d31a4eff7 HDDS-1753. Datanode unable to find chunk while replication data using ratis. (#1318) 2019-08-28 10:54:44 +05:30
Vivek Ratnavel Subramanian
b1eee8b52e HDDS-1946. CertificateClient should not persist keys/certs to ozone.m… (#1311) 2019-08-27 17:29:27 -07:00
Nanda kumar
2b9cc7eb95 HDDS-2037. Fix hadoop version in pom.ozone.xml. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-27 16:14:01 -07:00
Doroszlai, Attila
0883ce1021 HDDS-2026. Overlapping chunk region cannot be read concurrently 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-27 15:34:58 -07:00
Shashikant Banerjee
66cfa482c4 HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee(#1226). 2019-08-27 23:38:43 +05:30
Shashikant Banerjee
ce8eb1283a Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee." 
This reverts commit 62445021d5 as it has unintended changes in DirectoryWithSnapshotFeature class..
 2019-08-27 23:23:44 +05:30
Lokesh Jain
4379370fb1
HDDS-1981: Datanode should sync db when container is moved to CLOSED or QUASI_CLOSED state (#1319) 2019-08-27 10:22:03 +05:30
Nanda kumar
b661dcf563 HDDS-2002. Update documentation for 0.4.1 release. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-23 19:04:13 -07:00
Vivek Ratnavel Subramanian
d3fe993e60
HDDS-2023. Fix rat check failures in trunk (addendum) 2019-08-23 12:31:42 +02:00
Vivek Ratnavel Subramanian
e2a55482ee
HDDS-2023. Fix rat check failures in trunk 
Closes #1342
 2019-08-23 12:02:45 +02:00
Vivek Ratnavel Subramanian
b4a95a2b00
HDDS-2000. Don't depend on bootstrap/jquery versions from hadoop-trunk snapshot 
Closes #1339
 2019-08-23 10:56:00 +02:00
Bharat Viswanadham
4028cac56d
HDDS-1347. In OM HA getS3Secret call Should happen only leader OM. (#670) 2019-08-22 15:00:17 -07:00
avijayanhwx
28fb4b527a HDDS-2008 : Wrong package for RatisHelper class in hadoop-hdds/common module. (#1333) 2019-08-22 10:57:22 -07:00
Doroszlai, Attila
2ae7f444bd HDDS-1999. Basic acceptance test and SCM/OM web UI broken by Bootstrap upgrade (#1327) 2019-08-21 12:16:14 -07:00
Shashikant Banerjee
62445021d5 HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee. 2019-08-20 14:37:01 +05:30
avijayanhwx
d69a1a0aa4 HDDS-1105 : Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager (#1259) 2019-08-19 10:47:49 -07:00
Siyao Meng
12c7084be3 HDDS-1971. Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified (#1306) 2019-08-18 18:29:52 -07:00
avijayanhwx
e32f52c75f HDDS-1903 : Use dynamic ports for SCM in TestSCMClientProtocolServer … (#1303) 2019-08-18 10:08:38 -07:00
Vivek Ratnavel Subramanian
e61825682a HDDS-1979. Fix checkstyle errors (#1312) 2019-08-17 21:53:12 -07:00
HUAN-PING SU
02015e3ed8 HDDS-1959. Decrement purge interval for Ratis logs in datanode (#1301) 2019-08-17 16:54:33 +05:30
Li Cheng
bf3751521b HDDS-1894. Add filter to scmcli listPipelines. (#1286) 2019-08-16 14:33:04 +08:00
Doroszlai, Attila
77d102ceba HDDS-1910. Cannot build hadoop-hdds-config from scratch in IDEA 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-15 11:03:41 -07:00
dineshchitlangia
c801f7a26c HDDS-1768. Audit xxxAcl methods in OzoneManager (#1204) 2019-08-15 08:43:47 -07:00
Aravindan Vijayan
fc229b6490 HDDS-1832 : Improve logging for PipelineActions handling in SCM and datanode. (Change to Error logging) 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-13 22:04:31 -07:00
Siyao Meng
bf457797f6 HDDS-1920. Place ozone.om.address config key default value in ozone-site.xml 
Change-Id: Ic5970b383357147b74a01680aedf40bed4d3e176
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-13 20:27:37 -07:00
Elek, Márton
50a22b66c0 HDDS-1659. Define the process to add proposal/design docs to the Ozone subproject (#950) 
* HDDS-1659. Define the process to add proposal/design docs to the Ozone subproject
     * Remove Site improvements to display proposals
     * adding license header
     * clarify the support of the existing method
 2019-08-13 17:10:36 -07:00
Nanda kumar
69b74e9016 HDDS-1488. Scm cli command to start/stop replication manager. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-13 14:57:05 -07:00
dchitlangia
689a80d3ce HDDS-1886. Use ArrayList#clear to address audit failure scenario 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-08-13 14:37:32 -07:00
Siyao Meng
14a4ce3cee HDDS-1829 On OM reload/restart OmMetrics#numKeys should be updated. Contributed by Siyao Meng. 2019-08-08 13:38:10 -07:00
Nanda kumar
397a5633af
HDDS-1888. Add containers to node2container map in SCM as part of ICR processing. 
Signed-off-by: Nanda kumar <nanda@apache.org>
 2019-08-08 15:22:03 +05:30
Bharat Viswanadham
70b4617cfe
HDDS-1900. Remove UpdateBucket handler which supports add/remove Acl. (#1219) 2019-08-07 17:07:40 -07:00
Sammi Chen
8f9245bc2d HDDS-1865. Use "ozone.network.topology.aware.read" to control both RPC client and server side logic (#1184) 2019-08-07 16:59:13 -07:00
Doroszlai, Attila
0520f5cede
HDDS-1924. ozone sh bucket path command does not exist 
Closes #1245
 2019-08-07 18:21:37 +02:00
Xiaoyu Yao
a63023f261
HDDS-1901. Fix Ozone HTTP WebConsole Authentication. Contributed by Xiaoyu Yao. (#1228) 2019-08-06 12:07:53 -07:00
Nanda kumar
2662e20580
HDDS-1905. PipelineActionHandler is not closing the pipeline when close action is received. (#1227) 2019-08-06 14:36:04 +05:30
supratimdeka
f8ea6e1ce1 HDDS-1798. Propagate failure in writeStateMachineData to Ratis. Contributed by Supratim Deka (#1113) 2019-08-05 13:43:41 +05:30
Xiaoyu Yao
065cbc6b54 HDDS-1896. Suppress WARN log from NetworkTopology#getDistanceCost. Contributed by Xiaoyu Yao. (#1214) 2019-08-03 22:33:01 -07:00
Doroszlai, Attila
f4df97fd89 HDDS-1870. ConcurrentModification at PrometheusMetricsSink (#1179) 2019-08-03 22:26:25 -07:00
Vivek Ratnavel Subramanian
ec1d453846 HDDS-1788. Fix kerberos principal error in Ozone Recon. (#1201) 2019-08-03 10:49:07 -07:00
avijayanhwx
6e8c8413ee HDDS-1832 : Improve logging for PipelineActions handling in SCM and datanode. (#1200) 2019-08-02 22:31:00 +05:30
Doroszlai, Attila
946e301731 HDDS-1878. checkstyle error in ContainerStateMachine (#1195) 2019-08-02 22:27:54 +05:30
Sammi Chen
e111789aeb HDDS-1882. TestReplicationManager failed with NPE. (#1197) 2019-08-02 01:03:57 +05:30
Akira Ajisaka
8bda91d20a
HADOOP-16398. Exports Hadoop metrics to Prometheus (#1170) 2019-07-31 10:11:36 -07:00
Lokesh Jain
d4ab9aea6f HDDS-1816: ContainerStateMachine should limit number of pending apply transactions. Adds a config, uses snapshot threshold default value. (#1150) 2019-07-31 13:26:24 +05:30
Bharat Viswanadham
68f9f2d833
Revert HDDS-1829 On OM reload/restart OmMetrics#numKeys should be updated. (#1183) 2019-07-29 16:44:48 -07:00
Siyao Meng
4d07134b13 HDDS-1829 On OM reload/restart OmMetrics#numKeys should be updated (#1164) 2019-07-29 16:42:00 -07:00
avijayanhwx
60325c9611 HDDS-1391 : Add ability in OM to serve delta updates through an API. (#1033) 2019-07-29 09:39:40 -07:00
Eric Yang
d023663e3e HDDS-1833. Moved RefCountedDB stacktrace to log level trace. 
Contributed by Siddharth Wagle
 2019-07-29 12:05:24 -04:00
Márton Elek
b039f7591f
HDDS-1682. TestEventWatcher.testMetrics is flaky 
Closes #962.
 2019-07-29 11:04:45 +02:00
Doroszlai, Attila
902ff4a2f6
HDDS-1867. Invalid Prometheus metric name from JvmMetrics 
Closes #1172
 2019-07-29 10:27:55 +02:00
Chen, Junjie
c7c7a889a8 HDDS-1839: Change topology sorting related logs in Pipeline from INFO to DEBUG. Contributed by Junjie Chen. 
This closes #1158.
 2019-07-26 08:14:24 -07:00
Bharat Viswanadham
3426777140
HDDS-1861. Fix TableCacheImpl cleanup logic. (#1165) 2019-07-25 22:30:06 -07:00
Nanda kumar
2b1d8aedbb HDDS-1850. ReplicationManager should consider inflight replication and deletion while picking datanode for re-replication. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-25 14:25:39 -07:00
Xiaoyu Yao
b41ef61ebc HDDS-1858. mTLS support for Ozone is not correct. Contributed by Siddharth Wagle. 2019-07-25 09:52:02 -07:00
avijayanhwx
ba43233451 HDDS-1749 : Ozone Client should randomize the list of nodes in pipeline for reads. (#1124) 2019-07-25 20:24:46 +05:30
Nanda kumar
2546e6ece2 HDDS-1817. GetKey fails with IllegalArgumentException. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-24 15:59:30 -07:00
Bharat Viswanadham
fbe2fc6580 HDDS-1846. Default value for checksum bytes is different in ozone-site.xml and code. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-23 10:42:16 -07:00
Doroszlai, Attila
278390f5bf HDDS-1710. Fix checkstyle errors 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-23 07:56:30 -07:00
Márton Elek
c533b79c32 HDDS-1710. Publish JVM metrics via Hadoop metrics 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-22 16:27:50 -07:00
Doroszlai, Attila
c958eddcf4 HDDS-1811. Prometheus metrics are broken. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-22 14:12:06 -07:00
Hanisha Koneru
cdc36fe286 HDDS-1649. On installSnapshot notification from OM leader, download checkpoint and reload OM state (#948) 2019-07-22 12:06:55 -07:00
Sammi Chen
69a46a95bb HDDS-1713. ReplicationManager fail to find proper node topology based… (#1112) 2019-07-19 15:45:26 -07:00
Xiaoyu Yao
4e66cb9333 HDDS-1653. Add option to "ozone scmcli printTopology" to order the output acccording to topology layer. Contributed by Xiaoyu Yao. (#1067) 
* HDDS-1653. Add option to "ozone scmcli printTopology" to order the output acccording to topology layer. Contributed by Xiaoyu Yao.

* use ip/hostname instead of network name for -o output and add smoke test
 2019-07-19 11:00:49 +08:00
Doroszlai, Attila
d5ef38b093 HDDS-1822. NPE in SCMCommonPolicy.chooseDatanodes (#1120) 2019-07-18 13:28:03 -07:00
Shashikant Banerjee
20cf50c6d0 HDDS-1654. Ensure container state on datanode gets synced to disk whennever state change happens. Cotributed by Shashikant Banerjee. (#923) 2019-07-18 17:09:05 +05:30
Shashikant Banerjee
ccceedb432 HDDS-1780. TestFailureHandlingByClient tests are flaky. Contributed by Shashikant Banerjee. (#1073) 2019-07-18 16:01:58 +05:30
Lokesh Jain
23e9bebe13
HDDS-1767: ContainerStateMachine should have its own executors for executing applyTransaction calls (#1087) 2019-07-18 14:48:13 +05:30
Bharat Viswanadham
3dc256ef81
HDDS-1689. Implement S3 Create Bucket request to use Cache and DoubleBuffer. (#1088) 2019-07-17 21:46:27 -07:00
Bharat Viswanadham
73e6ffce69
HDDS-1802. Add Eviction policy for table cache. (#1100) 2019-07-17 18:19:06 -07:00
avijayanhwx
303a7f8a39 HDDS-1721 : Client Metrics are not being pushed to the configured sink while running a hadoop command to write to Ozone. (#1034) 2019-07-17 12:05:43 -07:00
Xiaoyu Yao
eb0849b547 HDDS-1801. Make Topology Aware Replication/Read non-default for ozone 0.4.1. Contributed by Xiaoyu Yao. 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-07-17 10:32:59 -07:00
Sammi Chen
493b0b5760 HDDS-1787. NPE thrown while trying to find DN closest to client. Contributed by Sammi Chen. (#1094) 2019-07-16 12:52:29 -07:00
Shashikant Banerjee
d21eccf8ba HDDS-1492. Generated chunk size name too long. Contributed by Shashikannt Banerjee. (#1084) 2019-07-16 18:01:13 +05:30
Nanda kumar
1502e6a22f HDDS-1756. DeleteContainerCommandHandler fails with NPE. Contributed by Nanda kumar(#1095). 2019-07-16 15:06:41 +05:30
Anu Engineer
47345e598e
Merge pull request #1058 from elek/HDDS-1763 
HDDS-1763. Use vendor neutral s3 logo in ozone doc
 2019-07-15 13:17:21 -07:00
Mukul Kumar Singh
0976f6fc30
HDDS-1766. ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed by Mukul Kumar Singh. (#1072) 2019-07-14 10:53:51 +05:30
Márton Elek
9119ed07ff
HDDS-1384. TestBlockOutputStreamWithFailures is failing 
Closes #1029
 2019-07-12 11:00:03 +02:00
Hrishikesh Gadre
14c43f85de HDDS-1752. ConcurrentModificationException while handling DeadNodeHandler event. (#1080) 2019-07-12 14:15:48 +05:30
supratimdeka
738fab3bff HDDS-1754. getContainerWithPipeline fails with PipelineNotFoundException. Contributed by Supratim Deka (#1081) 2019-07-12 10:31:48 +05:30
Nanda kumar
0f399b0d57
HDDS-1784. Missing HostName and IpAddress in the response of register command. 2019-07-11 19:01:06 +05:30
Hrishikesh Gadre
acef5e0cec HDDS-1201. Reporting corrupted containers info to SCM (#1032) 2019-07-11 14:49:58 +05:30
Ajay Yadav
cdb20adfcc HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. (#973) 2019-07-10 11:03:58 -07:00
supratimdeka
e0fa1040af HDDS-1748. Error message for 3 way commit failure is not verbose. Contributed by Supratim Deka (#1051) 2019-07-10 16:45:55 +05:30
supratimdeka
ac7a8accdf HDDS-1603. Handle Ratis Append Failure in Container State Machine. Contributed by Supratim Deka (#1019) 2019-07-10 07:49:36 +05:30
Xiaoyu Yao
030307226a
HDDS-1586. Allow Ozone RPC client to read with topology awareness. Contributed by Sammi Chen. 2019-07-09 14:43:55 -07:00
Mukul Kumar Singh
96d0555913 HDDS-1718. Increase Ratis Leader election timeout default. Contributed by Aravindan Vijayan & Siddharth Wagle. (#1062) 2019-07-09 23:17:50 +05:30
Vivek Ratnavel Subramanian
82d88a8d30 HDDS-1705. Recon: Add estimatedTotalCount to the response of containers and containers/{id} endpoints. Contributed by Vivek Ratnavel Subramanian. 2019-07-08 21:06:50 -07:00
Mukul Kumar Singh
e5ffb88257 HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh. (#1050) 
* HDDS-1550. MiniOzoneCluster is not shutting down all the threads during shutdown. Contributed by Mukul Kumar Singh.
 2019-07-08 20:24:12 -07:00
Lokesh Jain
b5d30e4914 HDDS-1750. Add block allocation metrics for pipelines in SCM. Contributed by Lokesh Jain. (#1047) 2019-07-09 08:42:55 +05:30
Mukul Kumar Singh
ce91d35b29
HDDS-1728. Add metrics for leader's latency in ContainerStateMachine. Contributed by Mukul Kumar Singh. (#1022) 2019-07-08 12:19:16 +05:30
Márton Elek
68f53f95ad
HDDS-1763. Use vendor neutral s3 logo in ozone doc. Contributed by Elek, Marton. 2019-07-04 17:14:19 +02:00
Mukul Kumar Singh
8b0d1adf31 HDDS-1616. ManagedChannel references are being leaked in while removing RaftGroup. Contributed by Mukul Kumar Singh. (#1039) 2019-07-02 14:34:49 -07:00
S O'Donnell
f8d62a9c4c
HDDS-1258. Fix error propagation for SCM protocol 
Closes #1001
 2019-07-01 20:04:32 +02:00
dineshchitlangia
1f75660350 HDDS-373. Genconf tool must generate ozone-site.xml with sample values (#1025) 2019-07-01 08:58:47 -07:00
Mukul Kumar Singh
9df6275954 HDDS-1555. Disable install snapshot for ContainerStateMachine. Contributed by Siddharth Wagle. (#846) 2019-07-01 20:52:32 +05:30
dineshchitlangia
da568996af HDDS-1733. Fix Ozone documentation (#1031) 2019-06-28 15:59:33 -07:00