Commit Graph

7643 Commits

Author SHA1 Message Date
Mukul Kumar Singh
979bbb4019 HDFS-13423. Ozone: Clean-up of ozone related change from hadoop-hdfs-project. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
25f2398bbd HDFS-13395. Ozone: Plugins support in HDSL Datanode Service. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
0ec88ea42b HDFS-13301. Ozone: Remove containerPort, ratisPort and ozoneRestPort from DatanodeID and DatanodeIDProto. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
61651dcf5c HDFS-13108. Ozone: OzoneFileSystem: Simplified url schema for Ozone File System. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar
9a914126a7 HDFS-13008. Ozone: Add DN container open/close state to container report. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Nanda kumar
f4466402ae HDFS-12965. Ozone: Documentation : Add ksm -createObjectStore command documentation. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Nanda kumar
1094af072c HDFS-11699. Ozone:SCM: Add support for close containers in SCM. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Nanda kumar
b4a3cf1476 HDFS-13070. Ozone: SCM: Support for container replica reconciliation - 1. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
4c10a849e8 HDFS-13078. Ozone: Update Ratis on Ozone to 0.1.1-alpha-8fd74ed-SNAPSHOT.
To fix large chunk reads (>4M) from Datanodes. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
136b419594 HDFS-13149. Ozone: Rename Corona to Freon. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
9272e1021d HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
377b31ffa1 HDFS-13022. Block Storage: Kubernetes dynamic persistent volume provisioner. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
ee5495456e HDFS-13116. Ozone: Refactor Pipeline to have transport and container specific information. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer
3cf07b43bc HDFS-13127. Fix TestContainerStateManager and TestOzoneConfigurationFields. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer
443425a5d9 HDFS-12522. Ozone: Remove the Priority Queues used in the Container State Manager. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Nanda kumar
32245c78e2 HDFS-13080. Ozone: Make finalhash in ContainerInfo of StorageContainerDatanodeProtocol.proto optional. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
150c4e1d0b HDFS-12879. Ozone : add scm init command to document. Contributed by Rahul Pathak. 2018-04-26 05:36:04 -07:00
Nanda kumar
7b3179f551 HDFS-13072. Ozone: DatanodeStateMachine: Handling Uncaught Exception in command handler thread. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
6b03fca37d HDFS-12868. Ozone: Service Discovery API. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
fb09d75379 HDFS-13018. Block Storage: make the iscsi target addres configurable for discovery. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
4bb9ad8e8f HDFS-13017. Block Storage: implement simple iscsi discovery in jscsi server. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
ba4d5a52a8 HDFS-12986. Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT). Contributed by Lokesh Jain 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
94c0346f35 HDFS-13024. Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Nanda kumar
4a051ba494 HDFS-12940. Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
1fb1ce107f HDFS-13012. TestOzoneConfigurationFields fails due to missing configs in ozone-default.xml. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Anu Engineer
916f31f7d1 HDFS-13026. Ozone: TestContainerPersistence is failing becaue of container data mismatch. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
28b87af51b HDFS-13013. Fix closeContainer API with the right container state change. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Anu Engineer
c4b88454a7 HDFS-12843. Ozone: Client: TestOzoneRpcClient#testPutKeyRatisThreeNodes is failing. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
98d62e55c4 Revert "HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee."
This reverts commit 6ce5ec676164b84a9e2f8dc65b5f2199a141506d.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
98eeabbdb3 HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
0c48cb2cf2 HFDS-13007. Ozone: Fix TestKeySpaceManager#testKSMInitializationFailure. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
98973cd0c4 HDFS-12966. Ozone: owner name should be set properly when the container allocation happens. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
5e31b920f0 HDFS-12871. Ozone: Service Discovery: Adding REST server details in ServiceList. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
3965f1ec99 HDFS-12853. Ozone: Optimize chunk writes for Ratis by avoiding double writes. Contributed by Mukul Kumar Singh 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
5cc5149bd8 HDFS-12961. Ozone: Use relative resource urls in KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
5766792cad HDFS-12870. Ozone: Service Discovery: REST endpoint in KSM for getServiceList. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
5f10c2d8ce HDFS-12980. Ozone: SCM: Restructuring container state transition and event. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
87b7774488 HDFS-12958. Ozone: remove setAllocatedBytes method in ContainerInfo. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Nanda kumar
ce19f09bd0 HDFS-12908. Ozone: write chunk call fails because of Metrics registry exception. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Nanda kumar
4b9f66a19e HDFS-12869. Ozone: Service Discovery: RPC endpoint in KSM for getServiceList. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
79432ee6cf HDFS-12863. Ozone: Avoid configuration.get() call for each container report. Contributed by Akira Ajisaka. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
0380dbb928 HDFS-12941. Ozone: ConfServlet does not trim values during the description parsing. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar
4dae68eebf HDFS-12741. Ozone: KSM: ADD support for KSM --createObjectStore command. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Chen Liang
caeeb78ca4 HDFS-12799. Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
966853894f HDFS-12795. Ozone: SCM: Support for Container LifeCycleState PENDING_CLOSE and LifeCycleEvent FULL_CONTAINER. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
26e270b908 HDFS-12751. Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer
fd09c2ce5b HDFS-12890 . Ozone: XceiverClient should have upper bound on async requests. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
2f3068bc72 HDFS-12698. Ozone: Use time units in the Ozone configuration values. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
8ff98e2af3 HDFS-12000. Ozone: Container : Add key versioning support-1. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
c0c87dea9b HDFS-12626. Ozone : delete open key entries that will no longer be closed. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Anu Engineer
6c630d0b09 HDFS-12899. Ozone: SCM: BlockManagerImpl close is called twice during StorageContainerManager#stop. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
74362cb503 HDFS-12901. Ozone: SCM: Expose StorageContainerManager#getScmId through container protocol. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
11473b8204 HDFS-12898. Ozone: TestSCMCli#testHelp and TestSCMCli#testNonExistCommand fail consistently. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
207523915d HDFS-12588. Use GenericOptionsParser for scm and ksm daemon. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar
7bba3943f0 HDFS-12876. Ozone: moving NodeType from OzoneConsts to Ozone.proto. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
85ec48a646 HDFS-12867. Ozone: TestOzoneConfigurationFields fails consistently. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer
538e17824d HDFS-12850. Ozone: TestContainerPersistence#testListContainer fails frequently due to timed out. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer
e32e187448 HDFS-12864. Ozone: Turn off full synced write in RocksDB MetadataStore. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
8023ba50b5 HDFS-12807. Ozone: Expose RockDB stats via JMX for Ozone metadata stores. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
86802a461c HDFS-12851. Ozone: Upgrade to latest ratis build. Contributed by Mukul Kumar Singh 2018-04-26 05:36:04 -07:00
Nanda kumar
ea86945754 HDFS-12842. Ozone: SCM: During startup SCM should check if the cluster is INITIALIZED. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
73ca8608da HDFS-12786. Ozone: add port/service names to the ksm/scm web ui. Contributed by Elek Marton. 2018-04-26 05:36:04 -07:00
Nanda kumar
f9c11d952c HDFS-12740. SCM should support a RPC to share the cluster Id with KSM and DataNodes. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
c8d8270f72 HDFS-12787. Ozone: SCM: Aggregate the metrics from all the container reports. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin
cbe4f314b3 HDFS-12731. Ozone: Fix inconsistent default value in ozone settings. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Anu Engineer
4deb845d7c HDFS-12759. Ozone: web: integrate configuration reader page to the SCM/KSM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
8e8ff85b16 HDFS-12830. Ozone: TestOzoneRpcClient#testPutKeyRatisThreeNodes fails. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin
214b52dfce HDFS-12500. Ozone: add logger for oz shell commands and move error stack traces to DEBUG level. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
419e86d580 HDFS-12805. Ozone: Redundant characters printed in exception log. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Anu Engineer
93f1b04949 HDFS-12798. Ozone: scm web: fix the node status table. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
4460ac9c57 HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Nanda kumar
446e84357d HDFS-12796. SCM should not start if Cluster Version file does not exist. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
740a06cdd7 HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Yiqun Lin
761926db2c HDFS-12793. Ozone : TestSCMCli is failing consistently. Contributed by Chen Liang. 2018-04-26 05:36:04 -07:00
Chen Liang
e395244e95 HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
cec96b296f HDFS-12549. Ozone: OzoneClient: Support for REST protocol. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
9734f505ea HDFS-12739. Add Support for SCM --init command. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
709d56fdc9 HDFS-12474. Ozone: SCM: Handling container report with key count and container usage. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
2ec24a7542 HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Yiqun Lin
132f30c8ea HDFS-12443. Ozone: Improve SCM block deletion throttling algorithm. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
7ebe79e879 HDFS-12720. Ozone: Ratis options are not passed from KSM Client protobuf helper correctly. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Weiwei Yang
0559265822 HDFS-12750. Ozone: Fix TestStorageContainerManager#testBlockDeletionTransactions. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao
0760418bec HDFS-12697. Ozone services must stay disabled in secure setup for alpha. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
b687902590 HDFS-12718. Ozone: fix thread number calculation in CBlockManager. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
76d34bca62 HDFS-12521. Ozone: SCM should read all Container info into memory when booting up. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer
6291ca1e32 HDFS-12663. Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Nanda kumar
80357c0eee HDFS-12678. Ozone: Corona: Add statistical information to json output. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer
1b56a73a4f HDFS-12680. Ozone: SCM: Lease support for container creation. Contributed by Nandakumar. 2018-04-26 05:36:04 -07:00
Anu Engineer
338c909ac6 HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
333dae888c HDFS-12690. Ozone: generate swagger descriptor for the Ozone REST Api. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
0582d8b1b1 Revert "HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton."
This reverts commit 18c0cdcec9ff5676b10ba711039143d07dd4a510.
2018-04-26 05:36:04 -07:00
Anu Engineer
402a1c1ba3 HDFS-12558. Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer
731a69faa1 HDFS-12693. Ozone: Enable XFrame options for KSM/SCM web ui. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Yiqun Lin
6783dad766 HDFS-11468. Ozone: SCM: Add Node Metrics for SCM. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Yiqun Lin
393a02d8e1 HDFS-12692. Ozone: fix javadoc/unit test for calling MetadataStore.getRangeKVs with non existing key. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Weiwei Yang
6d9f914b55 HDFS-12684. Ozone: SCMMXBean NodeCount is overlapping with NodeManagerMXBean. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Nanda kumar
cc86719451 HDFS-12675. Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00
Weiwei Yang
0f0417a12c HDFS-12673. Ozone: Log elapsed time for block deletion tasks. Contributed by Yiqun Lin. 2018-04-26 05:36:04 -07:00