Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
Yongjun Zhang
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
Erik Krogen
bad3ffd290
HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-02-22 13:58:49 -08:00
Erik Krogen
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
Chen Liang
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
Erik Krogen
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
Kitti Nanasi
0ab7fc9200
HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-31 16:07:24 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
Dinesh Chitlangia
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
Surendra Singh Lilhore
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
Inigo Goiri
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
Giovanni Matteo Fumarola
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
shwetayakkali@cloudera.com
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
Konstantin V Shvachko
bd2a59e508
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2018-12-24 09:39:20 -08:00
Konstantin V Shvachko
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
Chao Sun
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
Chen Liang
b8ad6c85a5
HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Chen Liang
71cf66e29b
HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
3bb92a1d9a
HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
2018-12-24 09:34:01 -08:00
Chen Liang
a3aab48df0
HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Chen Liang
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Chao Sun
1a9ba9616a
HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Erik Krogen
25b63e8da8
HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
Chen Liang
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Erik Krogen
aa42fb0db7
HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
f9fc01cd7f
HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
25d8e39b5c
HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
Chen Liang
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Erik Krogen
eae0a5d54a
HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Chao Sun
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
Konstantin V Shvachko
0105d7f597
HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Erik Krogen
a109f2b32f
HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
Erik Krogen
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Surendra Singh Lilhore
788e7473a4
HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.
2018-12-17 11:24:57 +05:30
Giovanni Matteo Fumarola
f858f18554
HDFS-14088. RequestHedgingProxyProvider can throw NullPointerException when failover due to no lock on currentUsedProxy. Contributed by Yuxuan Wang.
2018-12-14 11:04:36 -08:00
Zsolt Venczel
aa12859890
HDFS-14121. Log message about the old hosts file format is misleading
...
(Contributed by Zsolt Venczel via Daniel Templeton)
Change-Id: I7ff548f6c82e0aeb08a7a50ca7c2c827db8726bb
2018-12-14 13:54:57 +01:00
Daniel Templeton
4aa0609fb0
HDFS-13985. Clearer error message for ReplicaNotFoundException
...
(Contributed by Adam Antal via Daniel Templeton)
Change-Id: I68ae7a5bedecbc204c161fc3b3ce8878fe5d493d
2018-12-13 16:21:45 +01:00
Vinayakumar B
39dc7345b8
HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena.
2018-12-11 17:59:04 +05:30
Brahma Reddy Battula
f534736867
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
2018-11-30 00:18:27 +05:30
Brahma Reddy Battula
892b33e054
HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena.
2018-11-22 00:13:56 +05:30
Jason Lowe
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
Inigo Goiri
887244de4a
HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen.
2018-11-06 11:18:15 -08:00
Pranay Singh
d98b881e9a
HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-11-01 17:26:00 -07:00
Xiao Chen
db7e636824
HDFS-14027. DFSStripedOutputStream should implement both hsync methods.
2018-10-29 19:06:15 -07:00
Chen Liang
635786a511
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2018-10-23 14:53:45 -07:00
Inigo Goiri
88cce32551
HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin.
2018-10-19 10:38:04 -07:00
Xiao Chen
b6fc72a025
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
2018-10-15 10:51:55 -07:00
Vinayakumar B
b60ca37914
Fix potential FSImage corruption. Contributed by Daryn Sharp.
2018-10-15 15:48:26 +05:30
Xiao Chen
5ec86b445c
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
2018-10-12 09:35:52 -07:00
Hrishikesh Gadre
bf3d591f0c
HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:23:07 -07:00
Weiwei Yang
3ead525c71
HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:11:10 -07:00
Kitti Nanasi
10185d9a77
HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-10 08:55:32 -07:00
Hrishikesh Gadre
08bb6c49a5
HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads.
...
Contributed by Xiao Chen, Hrishikesh Gadre.
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-08 20:31:57 -07:00
Siyao Meng
396ce0d9f4
HDFS-13877. HttpFS: Implement GETSNAPSHOTDIFF. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-04 15:17:47 -07:00
Sunil G
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
Takanobu Asanuma
f6c5ef9903
HDFS-13943. [JDK10] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Akira Ajisaka.
2018-10-02 09:49:48 +09:00
Wei-Chiu Chuang
28ceb34a72
HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh.
2018-09-19 03:12:20 -07:00
Wei-Chiu Chuang
a1de8cbac5
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-09-14 05:22:56 +08:00
Vinayakumar B
30eceec342
HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena.
2018-09-10 09:10:51 +05:30
Giovanni Matteo Fumarola
eca1a4bfe9
HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering.
2018-09-06 14:48:00 -07:00
Kitti Nanasi
6e5ffb74dd
HDFS-13885. Add debug logs in dfsclient around decrypting EDEK. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-03 22:37:37 -07:00
Wei-Chiu Chuang
8aa6c4f079
Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
...
This reverts commit 26c2a97c56
.
2018-08-30 11:44:20 -07:00
Wei-Chiu Chuang
26c2a97c56
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-08-27 16:02:35 -07:00
Konstantin V Shvachko
a4121c71c2
HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
2018-08-24 18:27:30 -07:00
Xiao Chen
c67b0650ea
HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
2018-08-16 23:14:21 -07:00
Uma Maheswara Rao G
39ed3a66db
HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
Uma Maheswara Rao G
4402f3f855
HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.
2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
3b83110d5e
HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.
2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
c561cb316e
HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
68017e3349
HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
Uma Maheswara Rao G
6215e35bb6
HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu
2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
cd5262aba0
HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou
2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
b67ae6d9d7
HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu
2018-08-12 03:05:54 -07:00
Xiao Chen
9499df7b81
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
2018-08-08 10:40:20 -07:00
Daniel Templeton
849c45db18
HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
...
(Contributed by BELUGA BEHR via Daniel Templeton)
Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
2018-07-24 16:05:27 -07:00
Wei-Chiu Chuang
0247cb6318
HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
2018-07-06 14:59:49 -07:00
Andrew Wang
344f324710
HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
2018-07-03 11:07:45 +02:00
Inigo Goiri
1804a31515
HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-07-02 10:48:20 -07:00
Wei-Chiu Chuang
fef20a446f
HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
2018-07-02 09:41:15 -07:00
Andrew Wang
5d748bd056
HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
2018-07-02 12:11:06 +02:00
Andrew Wang
6ba9974108
HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
2018-07-02 12:02:19 +02:00
Todd Lipcon
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
Akira Ajisaka
fba9d7cd74
HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. Contributed by Takanobu Asanuma.
2018-06-18 10:17:07 -07:00
Chris Douglas
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
Inigo Goiri
2e5cfe6df3
HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
2018-06-11 11:20:29 -07:00
Arpit Agarwal
000a678396
HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.
2018-06-08 17:57:12 -07:00
Chao Sun
fba1c42adc
HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
2018-06-08 16:36:42 -07:00
Xiao Chen
774c1f199e
HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.
2018-06-05 21:25:08 -07:00
Xiao Chen
ba4011d64f
HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
2018-06-05 21:18:31 -07:00
Xiao Chen
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
Wei-Chiu Chuang
c95b9b5c07
HDFS-13440. Log HDFS file name when client fails to connect. Contributed by Gabor Bota.
2018-05-31 15:20:59 -07:00
Sean Mackrory
1361030e59
HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
2018-05-31 07:59:21 -06:00
Andrew Wang
c9b63deb53
HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
2018-05-24 09:56:23 -07:00
Bharat Viswanadham
bc6d9d4c79
HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
2018-05-23 10:15:40 -07:00
Sammi Chen
34e8b9f9a8
HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
2018-05-23 19:10:09 +08:00
Andrew Wang
1d2640b613
HDFS-13601. Optimize ByteString conversions in PBHelper.
2018-05-22 23:55:20 -07:00
Arpit Agarwal
0fc988e6a3
HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.
2018-05-16 11:28:39 -07:00
Xiao Chen
960940e0e0
HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure.
2018-05-14 09:28:39 -07:00
Ewan Higgs
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-27 19:21:37 +09:00
Lei Xu
a8e428b2dc
HDFS-13468. Add erasure coding metrics into ReadStatistics. (Contributed by Lei (Eddy) Xu)
2018-04-26 13:54:40 -07:00
Takanobu Asanuma
6f655aed3b
HADOOP-15404. Remove multibyte characters in DataNodeUsageReportUtil
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-26 13:54:40 -07:00
Nanda kumar
0ec88ea42b
HDFS-13301. Ozone: Remove containerPort, ratisPort and ozoneRestPort from DatanodeID and DatanodeIDProto. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Anu Engineer
4e61bc431e
HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
...
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
5e7164c614
HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
...
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Nanda kumar
9a914126a7
HDFS-13008. Ozone: Add DN container open/close state to container report. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Nanda kumar
b4a3cf1476
HDFS-13070. Ozone: SCM: Support for container replica reconciliation - 1. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
4c10a849e8
HDFS-13078. Ozone: Update Ratis on Ozone to 0.1.1-alpha-8fd74ed-SNAPSHOT.
...
To fix large chunk reads (>4M) from Datanodes. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
9272e1021d
HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
377b31ffa1
HDFS-13022. Block Storage: Kubernetes dynamic persistent volume provisioner. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
Anu Engineer
ee5495456e
HDFS-13116. Ozone: Refactor Pipeline to have transport and container specific information. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
3cf07b43bc
HDFS-13127. Fix TestContainerStateManager and TestOzoneConfigurationFields. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
443425a5d9
HDFS-12522. Ozone: Remove the Priority Queues used in the Container State Manager. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
6b03fca37d
HDFS-12868. Ozone: Service Discovery API. Contributed by Nanda Kumar.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
fb09d75379
HDFS-13018. Block Storage: make the iscsi target addres configurable for discovery. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
ba4d5a52a8
HDFS-12986. Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT). Contributed by Lokesh Jain
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
94c0346f35
HDFS-13024. Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
28b87af51b
HDFS-13013. Fix closeContainer API with the right container state change. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
98d62e55c4
Revert "HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee."
...
This reverts commit 6ce5ec676164b84a9e2f8dc65b5f2199a141506d.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
98eeabbdb3
HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
98973cd0c4
HDFS-12966. Ozone: owner name should be set properly when the container allocation happens. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
3965f1ec99
HDFS-12853. Ozone: Optimize chunk writes for Ratis by avoiding double writes. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
Anu Engineer
5766792cad
HDFS-12870. Ozone: Service Discovery: REST endpoint in KSM for getServiceList. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
5f10c2d8ce
HDFS-12980. Ozone: SCM: Restructuring container state transition and event. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Nanda kumar
87b7774488
HDFS-12958. Ozone: remove setAllocatedBytes method in ContainerInfo. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Nanda kumar
ce19f09bd0
HDFS-12908. Ozone: write chunk call fails because of Metrics registry exception. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Nanda kumar
4b9f66a19e
HDFS-12869. Ozone: Service Discovery: RPC endpoint in KSM for getServiceList. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
966853894f
HDFS-12795. Ozone: SCM: Support for Container LifeCycleState PENDING_CLOSE and LifeCycleEvent FULL_CONTAINER. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
26e270b908
HDFS-12751. Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Anu Engineer
fd09c2ce5b
HDFS-12890 . Ozone: XceiverClient should have upper bound on async requests. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Anu Engineer
2f3068bc72
HDFS-12698. Ozone: Use time units in the Ozone configuration values. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
8ff98e2af3
HDFS-12000. Ozone: Container : Add key versioning support-1. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
c0c87dea9b
HDFS-12626. Ozone : delete open key entries that will no longer be closed. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Anu Engineer
74362cb503
HDFS-12901. Ozone: SCM: Expose StorageContainerManager#getScmId through container protocol. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Nanda kumar
7bba3943f0
HDFS-12876. Ozone: moving NodeType from OzoneConsts to Ozone.proto. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
8023ba50b5
HDFS-12807. Ozone: Expose RockDB stats via JMX for Ozone metadata stores. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
86802a461c
HDFS-12851. Ozone: Upgrade to latest ratis build. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
Nanda kumar
f9c11d952c
HDFS-12740. SCM should support a RPC to share the cluster Id with KSM and DataNodes. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Anu Engineer
4460ac9c57
HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
Anu Engineer
740a06cdd7
HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Chen Liang
e395244e95
HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
cec96b296f
HDFS-12549. Ozone: OzoneClient: Support for REST protocol. Contributed by Nanda Kumar.
2018-04-26 05:36:04 -07:00
Nanda kumar
9734f505ea
HDFS-12739. Add Support for SCM --init command. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
709d56fdc9
HDFS-12474. Ozone: SCM: Handling container report with key count and container usage. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
2ec24a7542
HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
7ebe79e879
HDFS-12720. Ozone: Ratis options are not passed from KSM Client protobuf helper correctly. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer
76d34bca62
HDFS-12521. Ozone: SCM should read all Container info into memory when booting up. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
Anu Engineer
6291ca1e32
HDFS-12663. Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
1b56a73a4f
HDFS-12680. Ozone: SCM: Lease support for container creation. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
a467017068
HDFS-12411. Ozone: Add container usage information to DN container report. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Nanda kumar
9bb516118d
HDFS-12490. Ozone: OzoneClient: Add creation/modification time information in OzoneVolume/OzoneBucket/OzoneKey. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Mukul Kumar Singh
c85d3b2d25
HDFS-12593. Ozone: update Ratis to the latest snapshot. Contributed by Tsz Wo Nicholas Sze.
2018-04-26 05:36:04 -07:00
Yiqun Lin
d19b4c8763
HDFS-12583. Ozone: Fix swallow exceptions which makes hard to debug failures. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
e3b51d9074
HDFS-12598. Ozone: Fix 3 node ratis replication in Ozone. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
Nandakumar
3504af9411
HDFS-12537. Ozone: Reduce key creation overhead in Corona. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
Nandakumar
9630621be7
HDFS-12610. Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Nandakumar
c3ef381011
HDFS-12543. Ozone : allow create key without specifying size. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Anu Engineer
c17521b1bd
HDFS-12387. Ozone: Support Ratis as a first class replication mechanism. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
Anu Engineer
0b3270a8de
HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
6a754caca0
HDFS-12568. Ozone: Cleanup the ozone-default.xml. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
05246e2b32
HDFS-12060. Ozone: OzoneClient: Add list calls in RpcClient. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
7151a9b604
HDFS-12488. Ozone: OzoneRestClient timeout is not configurable. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
Chen Liang
9530153f33
HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Yiqun Lin
6b10723ec3
HDFS-12525. Ozone: OzoneClient: Verify bucket/volume name in create calls. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
a1a3ba6529
HDFS-12454. Ozone : the sample ozone-site.xml in OzoneGettingStarted does not work. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Anu Engineer
2804435386
HDFS-12017. Ozone: Container: Move IPC port to 98xx range. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Chen Liang
9ff136bb02
HDFS-12210. Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Chen Liang
dddded0016
Revert "Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh." to fix commit message.
...
This reverts commit 087c69ba2434813044089f733581607d172efbd1.
2018-04-26 05:36:04 -07:00
Chen Liang
8052374e79
Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Weiwei Yang
fd1564b87e
HDFS-12506. Ozone: ListBucket is too slow. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
Tsz-Wo Nicholas Sze
bf6f0cd831
HDFS-11563. Ozone: enforce DependencyConvergence uniqueVersions.
2018-04-26 05:36:04 -07:00
Anu Engineer
27ec22e20d
HDFS-12517. Ozone: mvn package is failing with out skipshade. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
Weiwei Yang
3b290fe3bf
HDFS-12466. Ozone: KSM: Make ozone.ksm.address as mandatory property for client. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Weiwei Yang
c329d3b4b4
HDFS-12329. Ozone: Ratis: Readonly calls in XceiverClientRatis should use sendReadOnly. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
77d4b18700
HDFS-12385. Ozone: OzoneClient: Refactoring OzoneClient API. Contributed by Nadakumar.
2018-04-26 05:36:04 -07:00
Anu Engineer
fe4139bdc3
HDFS-11612. Ozone: Cleanup Checkstyle issues. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
Chen Liang
b127ecd1ba
Revert "HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin." as it uses class not part of JRE.
...
This reverts commit c29aff44cfc78a2839f15e8cde3b80f0a31c80dd.
2018-04-26 05:36:04 -07:00
Chen Liang
f66a522695
HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Weiwei Yang
1087ce4cd2
HDFS-11676. Ozone: SCM CLI: Implement close container command. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
Weiwei Yang
743be0d7c0
HDFS-12235. Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions. Contributed by Weiwei Yang and Yuanbo Liu.
2018-04-26 05:36:04 -07:00
Anu Engineer
91a8f134b7
HDFS-11834. Ozone: Fix TestArchive#testArchive. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Weiwei Yang
7b3e7eb565
HDFS-12354. Ozone: Shuffle container list for datanode BlockDeletingService. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Chen Liang
8654698d74
HDFS-12365. Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
Weiwei Yang
4d3b8d332d
HDFS-12283. Ozone: DeleteKey-5: Implement SCM DeletedBlockLog. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
02f8c2224f
HDFS-11888. Ozone: SCM: use state machine for open containers allocated for key/blocks. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Weiwei Yang
9e733215a8
HDFS-12327. Ozone: support setting timeout in background service. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Anu Engineer
bd33038bf4
HDFS-12159. Ozone: SCM: Add create replication pipeline RPC. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
795ba1b3b7
HDFS-12313. Ozone: SCM: move container/pipeline StateMachine to the right package. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
Anu Engineer
3a661b7f82
HDFS-12238. Ozone: Add valid trace ID check in sendCommandAsync. Contributed by Ajay Kumar.
2018-04-26 05:36:04 -07:00
Chen Liang
f9bce29dcc
HDFS-12290. Block Storage: Change dfs.cblock.jscsi.server.address default bind address to 0.0.0.0. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Weiwei Yang
25586607a1
HDFS-12230. Ozone: KSM: Add creation time field in bucket info. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
Anu Engineer
c9ea431267
HDFS-12281. Ozone: Ozone-default.xml has 3 properties that do not match the default Config value. Contributed by Ajay Yadav.
2018-04-26 05:36:04 -07:00
Weiwei Yang
639b4fb8a9
HDFS-12196. Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
bb9a4793d3
HDFS-12259. Ozone: OzoneClient: Refactor move ozone client from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
Xiaoyu Yao
7290324fc8
HDFS-12267. Ozone: XceiverClientManager should not close the connection if client holds the reference. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00