big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,346 Commits 2 Branches 2 Tags 264 MiB
fdd96e46d1
Commit Graph

4812 Commits

Author SHA1 Message Date
Santosh Marella
cdc5de6448 HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 13:01:39 -07:00
Inigo Goiri
4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Anu Engineer
101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
Inigo Goiri
e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri
944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
He Xiaoqiao
33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena
c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Eric Yang
d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Erik Krogen
55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Konstantin V Shvachko
c31b7b8d52 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:34:29 -07:00
Inigo Goiri
c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Inigo Goiri
1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Virajith Jalaparti
77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Inigo Goiri
389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Erik Krogen
5847e00143 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 2019-05-13 08:39:59 -07:00
Rakesh Radhakrishnan
9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore
69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Inigo Goiri
865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Sahil Takiar
4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00
Arpit Agarwal
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 2019-04-29 14:49:35 -07:00
Sean Mackrory
a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Erik Krogen
174b7d3126 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-04-22 13:29:56 -07:00
Inigo Goiri
5321235fe8 HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena. 2019-04-22 13:23:42 -07:00
Inigo Goiri
be6c8014e6 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 2019-04-16 10:34:31 -07:00
Sahil Takiar
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer. 
Author:    Sahil Takiar <stakiar@cloudera.com>
 2019-04-12 18:12:02 -07:00
Chen Liang
626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Stephen O'Donnell
d2637cb176 HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-04 04:07:18 -07:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Lokesh Jain
cf268114c9 HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-02 12:24:55 -07:00
Uma Maheswara Rao G
35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan
f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Stephen O'Donnell
3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Erik Krogen
55b3a718e9 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-03-19 08:14:49 -07:00
Stephen O'Donnell
34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen
1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Wei-Chiu Chuang
6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri
945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B
f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Wei-Chiu Chuang
387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00
Wei-Chiu Chuang
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng." 
This reverts commit 5c10630ad8.
 2019-03-04 16:59:20 +09:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri
05e4ddeee6 HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena. 2019-02-28 10:03:05 -08:00
Erik Krogen
0feba4396f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-02-28 08:48:10 -08:00
Konstantin V Shvachko
a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Vinayakumar B
f4ae00c530 HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena." 2019-02-26 00:36:14 +05:30
Surendra Singh Lilhore
7d3b567194 HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi. 2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 2019-02-21 20:36:34 +05:30
Shweta Yakkali
1bea785020 HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 14:28:37 -08:00
Kitti Nanasi
14282e311b HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-19 12:04:56 -08:00
Inigo Goiri
1de25d134f HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR. 2019-02-18 15:30:38 -08:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Inigo Goiri
dde0ab55aa HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR. 2019-02-15 16:32:27 -08:00
Erik Krogen
64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5 HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena. 2019-02-14 22:13:14 +05:30
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Surendra Singh Lilhore
35d4f32b32 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 2019-02-13 23:06:46 +05:30
Vinayakumar B
7806403842 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:57:57 +05:30
Siyao Meng
5c10630ad8 HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-11 11:02:32 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Siyao Meng
6aa63452b3 HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-06 12:19:21 -08:00
Erik Krogen
d3de8e162b HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-02-06 09:54:47 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Inigo Goiri
0e79a86582 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 2019-02-04 11:59:48 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Wei-Chiu Chuang
d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00
Pranay Singh
1d523279da HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-25 09:02:57 -08:00
shwetayakkali@cloudera.com
f3e642d92b HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
 2019-01-23 15:57:39 -08:00
Kitti Nanasi
951cdd7e4c HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-23 14:40:57 -08:00
Surendra Singh Lilhore
221e308cb5 HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena. 2019-01-23 17:00:37 +05:30
Takanobu Asanuma
7d6792e5d2 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 2019-01-23 10:03:06 +09:00
Arpit Agarwal
1ff658b2ef HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 2019-01-21 20:44:37 -08:00
Inigo Goiri
f048512bb8 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 2019-01-16 10:14:22 -08:00
shwetayakkali@cloudera.com
4ab5260b7e HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
 2019-01-08 17:04:57 -08:00
Giovanni Matteo Fumarola
1a08302897 HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 2019-01-08 10:38:06 -08:00
Surendra Singh Lilhore
21fe77e3ab HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena. 2019-01-03 18:36:53 +05:30
Konstantin V Shvachko
c9d7737431 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
c3efc299fe HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
ef3e1929dd HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
cbc2f932eb HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Chen Liang
652b257478 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
b6f20c36c2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
1c4265d7bc HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
b5b9b77707 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Chen Liang
a65bb97f5d HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2018-12-24 09:34:00 -08:00
Erik Krogen
b74a7dbf88 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2018-12-24 09:34:00 -08:00
Erik Krogen
25b63e8da8 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
28820fb6b1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2018-12-24 09:34:00 -08:00
Chen Liang
d2e85b0b6d HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2018-12-24 09:34:00 -08:00
Erik Krogen
aa42fb0db7 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
25d8e39b5c HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Chen Liang
66ec3a67e8 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2018-12-24 09:34:00 -08:00
Erik Krogen
1e22f2bfbb HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2018-12-24 09:34:00 -08:00
Chao Sun
64b7cf59bd HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Konstantin V Shvachko
0105d7f597 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2018-12-24 09:33:59 -08:00
Erik Krogen
00e99c6594 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2018-12-24 09:33:59 -08:00
Erik Krogen
151c8ddbe4 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2018-12-24 09:33:59 -08:00
Erik Krogen
c81ac2ff02 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
e880660a20 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Erik Krogen
091ad974cd HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Erik Krogen
9bf0696c73 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
c9d73437e8 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Yiqun Lin
feb2664ac4 HDFS-13946. Log longest FSN write/read lock held stack trace. 2018-12-22 23:09:59 +08:00
Giovanni Matteo Fumarola
cb3382f667 HDFS-14145. TestBlockStorageMovementAttemptedItems#testNoBlockMovementAttemptFinishedReportAdded fails sporadically in Trunk. Contributed by Ayush Saxena. 2018-12-14 10:59:02 -08:00
Giovanni Matteo Fumarola
561b61cfb1 HDFS-14144. TestPred fails in Trunk. Contributed by Ayush Saxena. 2018-12-12 11:16:16 -08:00
Giovanni Matteo Fumarola
60af851e59 HDFS-14137. TestMaintenanceState fails with ArrayIndexOutOfBound Exception. Contributed by Ayush Saxena. 2018-12-11 11:22:32 -08:00
Vinayakumar B
39dc7345b8 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 2018-12-11 17:59:04 +05:30
Sean Mackrory
80e59e7876 HDFS-14101. Fixing underflow error in test. Contributed by Zsolt Venczel. 2018-12-10 14:03:08 -07:00
Inigo Goiri
8fc0d04517 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-12-07 18:30:36 -08:00
Vinayakumar B
c03024a530 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 2018-12-06 18:20:28 +05:30
Adam Antal
fb10803dfa HDFS-13818. Extend OIV to detect FSImage corruption. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-12-03 10:34:39 -08:00
Kitti Nanasi
dd5e7c6b72 HDFS-12946. Add a tool to check rack configuration against EC policies. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-12-03 10:01:09 -08:00
Surendra Singh Lilhore
042c8ef593 HDFS-14075. Terminate the namenode when failed to start log segment. Contributed by Ayush Saxena. 2018-12-02 12:31:08 +05:30
Brahma Reddy Battula
f534736867 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 2018-11-30 00:18:27 +05:30
Ewan Higgs
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.
 2018-11-29 15:12:17 +00:00
Vinayakumar B
b098281454 HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B. 2018-11-26 18:22:32 +05:30
Brahma Reddy Battula
892b33e054 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 2018-11-22 00:13:56 +05:30
Arpit Agarwal
1e15c7e855 HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel. 2018-11-15 12:42:31 -08:00
Inigo Goiri
460a94a10f HDFS-14045. Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN. Contributed by Jiandan Yang. 2018-11-15 10:58:57 -08:00
Inigo Goiri
993c2140cc HDFS-14054. TestLeaseRecovery2: testHardLeaseRecoveryAfterNameNodeRestart2 and testHardLeaseRecoveryWithRenameAfterNameNodeRestart are flaky. Contributed by Zsolt Venczel. 2018-11-15 09:29:14 -08:00
Xiao Chen
9da6054ca4 Revert "HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel." 
This reverts commit 7dc79a8b5b.
 2018-11-13 12:44:25 -08:00
Kitti Nanasi
8d99648c20 HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-11-08 10:01:19 -08:00
Inigo Goiri
887244de4a HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 2018-11-06 11:18:15 -08:00
Xiao Chen
ffc9c50e07 HDFS-14053. Provide ability for NN to re-replicate based on topology changes. Contributed by Hrishikesh Gadre. 2018-11-05 21:38:39 -08:00
Inigo Goiri
f3296501e0 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 2018-11-05 16:48:37 -08:00
Giovanni Matteo Fumarola
f3f5e7ad00 HDFS-14042. Fix NPE when PROVIDED storage is missing. Contributed by Virajith Jalaparti. 2018-11-05 11:02:31 -08:00
Xiao Chen
db7e636824 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 2018-10-29 19:06:15 -07:00
Wei-Chiu Chuang
63e7134df6 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fb2b72e6fc.
 2018-10-28 17:04:59 -07:00
Anu Engineer
4f10d7e23f HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.
 2018-10-26 16:41:23 -07:00
Inigo Goiri
1851d06eb3 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 2018-10-26 09:29:12 -07:00
Arpit Agarwal
9b899f1ebd HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 2018-10-25 15:14:13 -07:00
Wei-Chiu Chuang
fb2b72e6fc HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 2018-10-25 09:33:31 -07:00
Ajay Kumar
97bd49fc36 HDFS-14026. Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional. Contributed by Arpit Agarwal. 2018-10-24 21:55:42 -07:00
Inigo Goiri
936fc3f3c2 HDFS-14025. TestPendingReconstruction.testPendingAndInvalidate fails. Contributed by Ayush Saxena. 2018-10-24 16:18:43 -07:00
Inigo Goiri
c1874046e2 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 2018-10-24 09:20:38 -07:00
Chen Liang
635786a511 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2018-10-23 14:53:45 -07:00
Inigo Goiri
efdfe679d6 HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 2018-10-23 14:28:37 -07:00
Arpit Agarwal
c58811c77d HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 2018-10-22 14:44:28 -07:00
Inigo Goiri
f069d38c8d HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B. 2018-10-19 17:16:04 -07:00
Inigo Goiri
8b64fbab1a HDFS-13994. Improve DataNode BlockSender waitForMinLength. Contributed by BELUGA BEHR. 2018-10-19 10:36:03 -07:00
Inigo Goiri
be1cffb085 HDFS-14002. TestLayoutVersion#testNameNodeFeatureMinimumCompatibleLayoutVersions fails. Contributed by Takanobu Asanuma. 2018-10-18 15:25:53 -07:00
Xiao Chen
533138718c HDFS-13662. TestBlockReaderLocal#testStatisticsForErasureCodingRead is flaky 2018-10-16 19:33:55 -07:00
Inigo Goiri
f90c64e624 HDFS-13993. TestDataNodeVolumeFailure#testTolerateVolumeFailuresAfterAddingMoreVolumes is flaky. Contributed by Ayush Saxena. 2018-10-16 11:22:57 -07:00
Xiao Chen
b6fc72a025 HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 2018-10-15 10:51:55 -07:00
Vinayakumar B
b60ca37914 Fix potential FSImage corruption. Contributed by Daryn Sharp. 2018-10-15 15:48:26 +05:30
Vinayakumar B
6e0e6daaf3 HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena. 2018-10-12 17:34:10 +05:30
Hrishikesh Gadre
bf3d591f0c HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:23:07 -07:00
Weiwei Yang
3ead525c71 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:11:10 -07:00
Kitti Nanasi
10185d9a77 HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-10 08:55:32 -07:00
Inigo Goiri
605622c87b HDFS-11396. TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out. Contributed by Ayush Saxena. 2018-10-09 17:03:21 -07:00
Hrishikesh Gadre
08bb6c49a5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-08 20:31:57 -07:00
Inigo Goiri
73c660b43f HDFS-13958. Miscellaneous Improvements for FsVolumeSpi. Contributed by BELUGA BEHR. 2018-10-05 09:32:19 -07:00
Inigo Goiri
1dc0adfac0 HDFS-13947. Review of DirectoryScanner Class. Contributed by BELUGA BEHR. 2018-10-03 11:19:57 -07:00
Yiqun Lin
5689355783 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-02 09:43:14 +08:00
Takanobu Asanuma
e2113500df HDFS-8196. Post enabled Erasure Coding Policies on NameNode UI. Contributed by Kitti Nanasi and Kai Sasaki. 2018-09-29 00:53:35 +09:00
Akira Ajisaka
f9c0221623
HDFS-13932. [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions. Contributed by Andrew Purtell. 2018-09-27 11:44:42 +09:00
Brahma Reddy Battula
e0ff8e2c10 HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 2018-09-26 23:44:16 +05:30
Inigo Goiri
6275b42870 HDFS-13927. Addendum: Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-26 09:50:09 -07:00
Inigo Goiri
4758b4b6da HDFS-13927. Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-21 15:32:28 -07:00
Inigo Goiri
090272d7de HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 2018-09-19 13:00:30 -07:00
Inigo Goiri
56e0d635e0 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 2018-09-19 09:30:25 -07:00
Wei-Chiu Chuang
28ceb34a72 HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 2018-09-19 03:12:20 -07:00
Shweta
27978bcb66 HDFS-13833. Improve BlockPlacementPolicyDefault's consider load logic. Contributed by Shweta. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-18 20:23:50 -07:00
Wei-Chiu Chuang
a1de8cbac5 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-09-14 05:22:56 +08:00
Daniel Templeton
78bd3b1db9 HDFS-13846. Safe blocks counter is not decremented correctly if the block is striped 
(Contributed by Kitti Nanasi via Daniel Templeton)

Change-Id: Id41747a67dc946fdf0dbde90643bb1ea7e9e0f70
 2018-09-12 11:18:55 -07:00
Xiao Chen
8e54da1511 HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 2018-09-10 22:34:23 -07:00
Vinayakumar B
eef3bafae8 HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena. 2018-09-10 09:22:59 +05:30
Vinayakumar B
30eceec342 HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 2018-09-10 09:10:51 +05:30
Hrishikesh Gadre
335a8139f5 HDFS-13820. Add an ability to disable CacheReplicationMonitor. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-07 14:59:06 -07:00
Sean Mackrory
410dd3faa5 HDFS-13744. OIV tool should better handle control characters present in file or directory names. Contributed by Zsolt Venczel. 2018-09-07 12:39:33 -06:00
Giovanni Matteo Fumarola
eca1a4bfe9 HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering. 2018-09-06 14:48:00 -07:00
Wei-Chiu Chuang
8aa6c4f079 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 26c2a97c56.
 2018-08-30 11:44:20 -07:00
Shweta
33f42efc94 HDFS-13837. Enable debug log for LeaseRenewer in TestDistributedFileSystem. Contributed by Shweta. 2018-08-28 13:56:32 -07:00
Wei-Chiu Chuang
26c2a97c56 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-08-27 16:02:35 -07:00
Yiqun Lin
b9b964d253 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-27 14:55:46 +08:00
Surendra Singh Lilhore
96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Vinayakumar B
770d9d9bb6 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:33:19 +05:30
Giovanni Matteo Fumarola
79c97f6a0b HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He. 2018-08-17 14:40:00 -07:00
Xiao Chen
c67b0650ea HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 2018-08-16 23:14:21 -07:00
Wei-Chiu Chuang
1290e3c647 HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-16 16:29:38 -07:00
Daniel Templeton
8512e1a91b HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
 2018-08-16 15:00:45 -07:00
Xiao Chen
7dc79a8b5b HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-15 13:53:47 -07:00
Akira Ajisaka
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Daniel Templeton
4a5006b1d0 HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky 
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
 2018-08-14 17:22:15 -07:00
Wei-Chiu Chuang
363bd16e31 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 17:19:00 -07:00
Wei-Chiu Chuang
4023eeba05 HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 2018-08-13 17:36:13 -07:00
drankye
475bff6e8e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-13 17:32:56 +08:00
Uma Maheswara Rao Gangumalla
3ac07b720b HDFS-13808: [SPS]: Remove unwanted FSNamesystem #isFileOpenedForWrite() and #getFileInfo() function. Contributed by Rakesh R. 2018-08-12 03:06:07 -07:00
Uma Maheswara Rao G
39ed3a66db HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Uma Maheswara Rao G
66e8f9b315 HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
2acc50b826 HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Surendra Singh Lilhore
75ccc1396b HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Rakesh Radhakrishnan
8467ec24fb HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00