big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,689 Commits 2 Branches 2 Tags 264 MiB
e079fa6577
Commit Graph

9115 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface 
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
 2020-03-09 14:51:16 +00:00
Steve Loughran
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden 
Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
 2020-03-09 14:44:28 +00:00
Inigo Goiri
4062217189 HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena. 2020-03-06 10:14:53 -08:00
Inigo Goiri
3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Inigo Goiri
1d4d0fcbe1 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 2020-03-04 10:02:54 -08:00
Akshay Nehe
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871) 2020-03-04 11:07:31 +09:00
Xiaoyu Yao
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file 
Contributed by Xiaoyu Yao.

Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails

Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.

This is a convergence of a fix for S3 meeting an issue in HDFS.

S3: a HEAD against a file can cache a 404, 
 -you must not do any existence checks, including deleteOnExit(),
  until the file is written. 

Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct. 

HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.

This patch:

* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
 2020-03-02 13:22:00 +00:00
Ayush Saxena
1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Inigo Goiri
97b797c314 HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun. 2020-02-28 18:47:22 -08:00
Takanobu Asanuma
9a3a28018a HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina. 2020-02-29 10:30:26 +09:00
Surendra Singh Lilhore
fccfb02672 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:33:06 +05:30
Ayush Saxena
429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Istvan Fajth
57aa048516
HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 2020-02-27 07:48:14 -08:00
Surendra Singh Lilhore
2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena
033a3d7ff9 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:26 +05:30
Ayush Saxena
209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang
1c5d2f1fdc HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 2020-02-24 13:39:16 -08:00
Ayush Saxena
7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena
93b8f453b9 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:01 +05:30
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena
9eb7a8bdf8 HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi. 2020-02-23 14:25:39 +05:30
Ayush Saxena
ba9025c7cd HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun. 2020-02-22 14:27:26 +05:30
Konstantin V Shvachko
ecbcb058b8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 2020-02-21 17:53:37 -08:00
Inigo Goiri
ed70c115a8 HDFS-15172. Remove unnecessary deadNodeDetectInterval in DeadNodeDetector#checkDeadNodes(). Contributed by Lisheng Sun. 2020-02-21 16:36:30 -08:00
Konstantin V Shvachko
6f84269bcd HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 2020-02-21 10:51:14 -08:00
Masatake Iwasaki
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 2020-02-21 11:56:07 +09:00
Inigo Goiri
ec7507162c HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham. 2020-02-19 11:33:58 -08:00
Ayush Saxena
ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Ayush Saxena
84f7638840 HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri. 2020-02-18 00:43:33 +05:30
xiaoxiaopan118
439d935e1d
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 2020-02-17 15:55:10 +09:00
Surendra Singh Lilhore
810783d443 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-02-16 12:15:03 +05:30
Ayush Saxena
c75756fe13 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore
a98352ced1 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 16:57:41 +05:30
Akira Ajisaka
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router (#1832) 
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
 2020-02-13 10:06:07 +09:00
Ayush Saxena
f09710bbb8 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:35 +05:30
Ayush Saxena
3df0adaaea HDFS-15127. RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Contributed by Inigo Goiri 2020-02-12 19:41:04 +05:30
Kihwal Lee
9b8a78d97b HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.
 2020-02-11 12:40:00 -06:00
Stephen O'Donnell
d7c136b9ed HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-11 08:00:15 -08:00
Ayush Saxena
6191d4b4a0 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 2020-02-09 23:32:22 +05:30
Ayush Saxena
d23317b102 HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:33:57 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Chen Liang
ce7b8b5634 HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:12:35 -08:00
Kihwal Lee
10a60fbe20 HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 2020-02-04 12:22:35 -06:00
Stephen O'Donnell
1e3a0b0d93 HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2020-01-31 16:13:02 -08:00
Kihwal Lee
799d4c1cf4 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.
 2020-01-29 11:00:27 -06:00
Chen Liang
483397c7f7 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-28 15:20:36 -08:00
Chen Liang
3e86807802 Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang." 
This reverts commit ff8ff0f7e5.
 2020-01-28 15:19:47 -08:00
Chen Liang
ff8ff0f7e5 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang. 2020-01-28 15:14:58 -08:00
Inigo Goiri
1839c467f6 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 2020-01-28 10:10:35 -08:00
Inigo Goiri
061421fc6d HDFS-15145. HttpFS: getAclStatus() returns permission as null. Contributed by hemanthboyina. 2020-01-28 10:04:38 -08:00
Ayush Saxena
87c198468b HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:09:38 +05:30
Ayush Saxena
f876dc228b HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena. 2020-01-28 19:34:21 +05:30
Ayush Saxena
6d008c0d39 HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest 2020-01-25 11:47:29 +05:30
Kihwal Lee
d10f77e3c9 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.
 2020-01-24 09:15:27 -06:00
Ayush Saxena
92c58901d7 HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena 2020-01-23 18:18:34 +05:30
Kihwal Lee
bd03053ea2 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 2020-01-22 09:14:38 -06:00
Inigo Goiri
8cfc3673dc HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 2020-01-21 13:29:20 -08:00
Inigo Goiri
b657822b98 HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein. 2020-01-21 13:22:53 -08:00
Akira Ajisaka
f6d20daf40
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 2020-01-21 18:03:24 +09:00
Akira Ajisaka
1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 2020-01-21 10:58:32 +09:00
Inigo Goiri
263413e838 HDFS-15112. RBF: Do not return FileNotFoundException when a subcluster is unavailable. 2020-01-16 10:39:14 -08:00
Vinayakumar B
edbbc03ce7
HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. (#1803) 2020-01-16 23:27:50 +05:30
Chao Sun
d7c4f8ab21 HDFS-13616. Batch listing of multiple directories (#1725) 2020-01-15 17:22:47 -08:00
Wei-Chiu Chuang
6b86a5110e HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 2020-01-13 10:48:53 -08:00
Vinayakumar B
52b360a928
HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. (#1801) 2020-01-13 10:40:29 +05:30
Ayush Saxena
cebce0a348 HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:29:11 +05:30
Ayush Saxena
7b62409ace HDFS-15108. RBF: MembershipNamenodeResolver should invalidate cache incase of active namenode update. Contributed by Ayush Saxena. 2020-01-11 10:21:31 +05:30
Chen Liang
26a969ec73 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 2020-01-10 17:06:59 -08:00
Kihwal Lee
5fb901ac40 HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 2020-01-10 12:43:21 -06:00
Takanobu Asanuma
9da294a140 HDFS-15110. HttpFS: post requests are not supported for path "/". Contributed by hemanthboyina. 2020-01-10 17:53:19 +09:00
Akira Ajisaka
0315ef8448 HDFS-15100. RBF: Print stacktrace when DFSRouter fails to fetch/parse JMX output from NameNode. (#1800) 2020-01-10 13:16:57 +09:00
Ayush Saxena
b32757c616 HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena. 2020-01-10 08:14:56 +05:30
Takanobu Asanuma
782c0556fb HDFS-15102. HttpFS: put requests are not supported for path "/". Contributed by hemanthboyina. 2020-01-10 09:52:13 +09:00
Surendra Singh Lilhore
bf45f3b80a HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:04:05 +05:30
Ayush Saxena
8fe01db34a HDFS-15094. RBF: Reuse ugi string in ConnectionPoolID. Contributed by Ayush Saxena. 2020-01-09 09:02:38 +05:30
Ayush Saxena
fd30f4c52b HDFS-15096. RBF: GetServerDefaults Should be Cached At Router. Contributed by Ayush Saxena. 2020-01-09 08:26:51 +05:30
Rakesh Radhakrishnan
7030722e5d HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 2020-01-08 14:25:17 +05:30
Masatake Iwasaki
aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 2020-01-08 16:45:39 +09:00
Masatake Iwasaki
a43c177f1d HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 2020-01-08 11:28:34 +09:00
Takanobu Asanuma
59aac00283 HDFS-15066. HttpFS: Implement setErasureCodingPolicy , unsetErasureCodingPolicy , getErasureCodingPolicy. Contributed by hemanthboyina. 2020-01-07 11:10:32 +09:00
Takanobu Asanuma
4a76ab777f HDFS-15090. RBF: MountPoint Listing Should Return Flag Values Of Destination. Contributed by Ayush Saxena. 2020-01-06 18:09:59 +09:00
luhuachao
77ae7b9ce2 HDFS-15089. RBF: SmallFix for RBFMetrics in doc (#1786) 2020-01-06 12:31:13 +09:00
Ayush Saxena
f8644fbe9f HDFS-15091. Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock. Contributed by Ayush Saxena. 2020-01-04 19:02:59 +05:30
Masatake Iwasaki
037ec8cfb1 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-01-04 01:55:36 +09:00
Rakesh Radhakrishnan
d79cce20ab HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 2020-01-02 11:44:00 +05:30
Takanobu Asanuma
074050ca59 HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina. 2020-01-01 11:26:38 +09:00
Ayush Saxena
62423910a4 HDFS-14937. [SBN read] ObserverReadProxyProvider should throw InterruptException. Contributed by xuzq. 2019-12-29 13:07:22 +05:30
Surendra Singh Lilhore
ee51eadda0 HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:15:54 +05:30
Takanobu Asanuma
dc32f583af HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 2019-12-28 21:32:15 +09:00
Ayush Saxena
8730a7bf60 HDFS-15003. RBF: Make Router support storage type quota. Contributed by Jinglun. 2019-12-27 09:22:58 +05:30
Ayush Saxena
80f91d14ab HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 2019-12-27 09:12:44 +05:30
Ayush Saxena
300505c562 HDFS-15054. Delete Snapshot not updating new modification time. Contributed by hemanthboyina. 2019-12-25 12:42:50 +05:30
Ayush Saxena
df622cf4a3 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 2019-12-25 11:07:25 +05:30
Chandra Sanivarapu
d8cd7098b4 HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 2019-12-25 10:50:01 +09:00
Konstantin V Shvachko
b98ac2a3af HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 2019-12-24 12:08:34 -08:00
Inigo Goiri
52d7b745c6 HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 2019-12-19 09:42:17 -08:00
Inigo Goiri
b86895485d HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 2019-12-19 09:34:43 -08:00
Shashikant Banerjee
fdd96e46d1 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 2019-12-18 22:50:46 +05:30
Inigo Goiri
24080666e5 HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun. 2019-12-16 18:41:45 -08:00
Masatake Iwasaki
dc6cf17b34 HDFS-15048. Fix findbug in DirectoryScanner. 2019-12-16 19:00:57 +09:00
Ayush Saxena
7a87007545 HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina. 2019-12-15 00:33:32 +05:30
Ayush Saxena
72aee114f8 HDFS-15053. RBF: Add permission check for safemode operation. Contributed by Xiaoqiao He. 2019-12-14 17:57:23 +05:30
Inigo Goiri
7fe924b1c0 HDFS-15016. RBF: getDatanodeReport() should return the latest update. Contributed by Inigo Goiri. 2019-12-13 10:51:14 -08:00
Chen Liang
65c4660bcd HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 2019-12-12 10:22:05 -08:00
Masatake Iwasaki
bdd00f10b4
HDFS-15047. Document the new decommission monitor (HDFS-14854). (#1755) 2019-12-13 00:35:41 +09:00
He Xiaoqiao
0e28cd8f63 HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-12 02:23:32 -08:00
Takanobu Asanuma
93bb368094 HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li 2019-12-12 10:56:19 +09:00
Erik Krogen
c174d50b30 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 2019-12-11 09:39:37 -08:00
Surendra Singh Lilhore
c2e9783d5f HDFS-15045. DataStreamer#createBlockOutputStream() should log exception in warn. Contributed by Ravuri Sushma Sree. 2019-12-11 08:52:55 +05:30
Stephen O'Donnell
c93cb6790e HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-10 17:16:43 -08:00
Takanobu Asanuma
c4733377d0
HDFS-15040. RBF: Secured Router should not run when SecretManager is not running. (#1745) 2019-12-10 16:03:32 +09:00
Akira Ajisaka
9f09852051
HDFS-15043. RBF: The detail of the Exception is not shown in ZKDelegationTokenSecretManagerImpl (#1748) 2019-12-10 16:00:20 +09:00
Masatake Iwasaki
4dffd81bb7
HDFS-14522. Allow compact property description in xml in httpfs. (#1737) 2019-12-10 10:30:57 +09:00
Masatake Iwasaki
11cd5b6e39 HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-12-07 08:46:45 +09:00
Wei-Chiu Chuang
ecd461f940 HDFS-14751. Synchronize on diffs in DirectoryScanner. Contributed by Lisheng Sun. 2019-12-06 15:10:40 -08:00
Wei-Chiu Chuang
313b76f8e9 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-12-06 15:10:40 -08:00
Ayush Saxena
705b172b95 HDFS-14998. [SBN read] Update Observer Namenode doc for ZKFC after HDFS-14130. Contributed by Fei Hui. 2019-12-06 23:04:09 +05:30
Ayush Saxena
83a14559e5 HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui. 2019-12-05 18:52:01 +05:30
Yiqun Lin
47fdae7904 HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun. 2019-12-04 18:19:11 +08:00
Ayush Saxena
0c217feed8 HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun. 2019-12-03 09:18:09 +05:30
Takanobu Asanuma
4ede8bce28 HDFS-9695. HTTPFS - CHECKACCESS operation missing. Contributed by hemanthboyina 2019-12-03 09:38:36 +09:00
Ayush Saxena
6b2d6d4aaf HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina. 2019-11-29 23:55:37 +05:30
Surendra Singh Lilhore
44f7b9159d HDFS-15013. Reduce NameNode overview tab response time. Contributed by HuangTao. 2019-11-29 00:42:31 +05:30
Surendra Singh Lilhore
0384687811 HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-28 23:49:35 +05:30
Ayush Saxena
46166bd8d1 HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena. 2019-11-28 18:28:20 +05:30
Yiqun Lin
c3659f8f94 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. 2019-11-28 14:41:49 +08:00
Yiqun Lin
2b452b4e60 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 2019-11-28 10:43:35 +08:00
Yiqun Lin
c8bef4d6a6 HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. 2019-11-27 10:57:20 +08:00
Ayush Saxena
049940e77b HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:30:02 +05:30
Ayush Saxena
c422e36397 HDFS-13842. RBF: Exceptions are conflicting when creating the same mount entry twice. Contributed by Ranith Sardar. 2019-11-22 11:42:19 +05:30
Takanobu Asanuma
b25e94ce29 HDFS-14924. RenameSnapshot not updating new modification time. Contributed by hemanthboyina 2019-11-22 13:48:02 +09:00
Yiqun Lin
9b6906fe91 HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun. 2019-11-22 10:53:55 +08:00
Ayush Saxena
b89fd4dfe9 HDFS-15002. RBF: Fix annotation in RouterAdmin. Contributed by Jinglun. 2019-11-22 01:08:11 +05:30
Surendra Singh Lilhore
26270196a2 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:31:25 +05:30
Ayush Saxena
98d249dcda HDFS-14996. RBF: GetFileStatus fails for directory with EC policy set in case of multiple destinations. Contributed by Ayush Saxena. 2019-11-21 12:08:42 +05:30
Inigo Goiri
3037762b2c HDFS-14949. Add getServerDefaults() support to HttpFS. Contributed by hemanthboyina. 2019-11-20 10:51:58 -08:00
Surendra Singh Lilhore
fd264b8265 HDFS-14995. Use log variable directly instead of passing as argument in InvalidateBlocks#printBlockDeletionTime(). Contributed by Lisheng Sun. 2019-11-21 00:00:42 +05:30
He Xiaoqiao
0b50aa29fd HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
 2019-11-19 12:16:57 -08:00
Surendra Singh Lilhore
c870514740 HDFS-14992. TestOfflineEditsViewer is failing in Trunk. Contributed by hemanthboyina. 2019-11-20 00:07:43 +05:30
Ayush Saxena
34cb595d2e HDFS-14967. TestWebHDFS fails in Windows. Contributed by Renukaprasad C. 2019-11-18 18:49:58 +05:30
Ayush Saxena
12617fad2e HDFS-14955. RBF: getQuotaUsage() on mount point should return global quota. Contributed by Jinglun. 2019-11-18 16:04:52 +05:30
Ayush Saxena
3b5a0e86c1 HDFS-14974. RBF: Make tests use free ports. Contributed by Inigo Goiri. 2019-11-18 15:49:36 +05:30
Yiqun Lin
b3119b9ab6 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. 2019-11-16 11:32:41 +08:00
Hui Fei
67f2c491fe HDFS-14802. The feature of protect directories should be used in RenameOp (#1669) 2019-11-15 12:35:37 -08:00
He Xiaoqiao
c892a879dd HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
 2019-11-15 12:16:31 -08:00
Erik Krogen
b2cc8b6b4a HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 2019-11-15 10:10:37 -08:00
Erik Krogen
586defe711 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 2019-11-13 11:46:35 -08:00
Chao Sun
97ec34e117 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 2019-11-12 11:35:50 -08:00
Inigo Goiri
40150da1e1 HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina. 2019-11-12 10:53:54 -08:00
Ayush Saxena
77934bc07b HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui. 2019-11-11 11:05:34 +05:30
Wang Yuxuan
b25a37c322 HDFS-14962. RBF: ConnectionPool#newConnection() error log wrong protocol class (#1699). Contributed by Yuxuan Wang. 2019-11-11 10:48:38 +05:30
Takanobu Asanuma
6663d6a5c2 HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li 2019-11-11 14:04:57 +09:00
Surendra Singh Lilhore
320008bb7c HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 2019-11-09 22:14:35 +05:30
Ayush Saxena
42fc8884ab HDFS-14815. RBF: Update the quota in MountTable when calling setQuota on a MountTable src. Contributed by Jinglun. 2019-11-08 08:48:52 +05:30
Ayush Saxena
247584eb63 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:49:01 +05:30
Chen Liang
dd900259c4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 09:56:19 -08:00
Chen Liang
9d0d580031 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 09:30:04 -08:00
Ayush Saxena
9e287054a8 HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 2019-11-06 22:02:03 +05:30
Ayush Saxena
dcf55838ae HDFS-14880. Correct the sequence of statistics & exit message in balencer. Contributed by Renukaprasad C. 2019-11-06 21:45:03 +05:30
Surendra Singh Lilhore
c36014165c HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 2019-11-06 19:28:55 +05:30
Inigo Goiri
bfb8f28cc9 HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang. 2019-11-05 09:19:52 -08:00
Takanobu Asanuma
7901062707 HDFS-14891. RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme. Contributed by Xieming Li 2019-11-05 15:03:09 +09:00
Ayush Saxena
b643a1cbe8 HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun. 2019-11-05 09:23:20 +05:30
Ayush Saxena
2ffec347eb HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-05 01:37:15 +05:30
Wei-Chiu Chuang
eb73ba6ed5 HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 2019-11-04 09:40:42 -08:00
Zhao Junwang
de6b8b0c0b HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
 2019-11-01 16:37:53 -07:00
Ayush Saxena
7d7acb004a HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:50:10 +05:30
Inigo Goiri
f18bbdd9d8 HDFS-14927. RBF: Add metrics for async callers thread pool. Contributed by Leon Gao. 2019-11-01 10:14:31 -07:00
Surendra Singh Lilhore
02009c3bb7 HDFS-14768. EC : Busy DN replica should be consider in live replica check. Contributed by guojh. 2019-11-01 22:15:05 +05:30
Ayush Saxena
9d25ae7669 HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:49:37 +05:30
Ayush Saxena
d9fbedc4ae HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun. 2019-10-31 23:39:27 +05:30
Jonathan Hung
e6137d0e50 Make upstream aware of 2.10.0 release 2019-10-30 20:58:32 -07:00
Ayush Saxena
fa4904cdca HDFS-14935. Refactor DFSNetworkTopology#isNodeInScope. Contributed by Lisheng Sun. 2019-10-29 12:30:24 +05:30
Eric Yang
30ed24a421 HDFS-14730. Removed unused configuration dfs.web.authentication.filter. 
Contributed by Chen Zhang
 2019-10-28 19:27:32 -04:00
Eric Badger
9ef6ed9c1c HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 2019-10-28 16:37:04 +00:00
zhaoyim
30db895b59 HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-25 13:09:14 -07:00
Takanobu Asanuma
862526530a HDFS-14933. Fixing a typo in documentation of Observer NameNode. Contributed by Xieming Li 2019-10-25 15:23:05 +09:00
Takanobu Asanuma
0db0f1e399 HDFS-14917. Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html". Contributed by Xieming Li 2019-10-25 10:27:38 +09:00
Wei-Chiu Chuang
a1b4eebcc9
HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2019-10-24 13:09:52 -07:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Vinayakumar B
ee699dc26c HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:14:09 +05:30
Mukul Kumar Singh
a901405ad8 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-22 13:31:15 -07:00
Ayush Saxena
6020505943 HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:44:22 +05:30
Ayush Saxena
19f35cfd57 HDFS-14918. Remove useless getRedundancyThread from BlockManagerTestUtil. Contributed by Fei Hui. 2019-10-22 18:34:02 +05:30
Wei-Chiu Chuang
72003b19bf HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-21 17:31:43 -07:00
Kihwal Lee
c4e27ef773 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 2019-10-21 16:33:15 -05:00
Ayush Saxena
74c2329fc3 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:35:12 +05:30
Ayush Saxena
ff6a492d66 HDFS-14916. RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'. Contributed by Xieming Li. 2019-10-21 18:20:00 +05:30
Ayush Saxena
447f46d962 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:10:59 +05:30
Inigo Goiri
e6f95eb0f7 HDFS-14887. RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable. Contributed by hemanthboyina. 2019-10-19 13:24:58 -07:00
Yuming Wang
3d41f33018 HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng. 
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-18 08:25:02 -07:00
Surendra Singh Lilhore
54dc6b7d72 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 2019-10-17 22:28:47 +05:30
Ayush Saxena
5527d79adb HDFS-14810. Review FSNameSystem editlog sync. Contributed by Xiaoqiao He. 2019-10-17 21:56:30 +05:30
Ayush Saxena
375224edeb HDFS-14739. RBF: LS command for mount point shows wrong owner and permission information. Contributed by Jinglun. 2019-10-16 19:26:01 +05:30
Surendra Singh Lilhore
336abbd873 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 2019-10-14 23:31:43 +05:30
Ashvin
fabd41fa48 HDFS-14856. Fetch file ACLs while mounting external store. (#1478) 2019-10-14 09:44:56 -07:00
Ayush Saxena
5f4641a120 HDFS-14238. A log in NNThroughputBenchmark should change log level to INFO instead of ERROR. Contributed by Shen Yinjie. 2019-10-12 21:06:23 +05:30
Ayush Saxena
6e5cd5273f HDFS-14899. Use Relative URLS in Hadoop HDFS RBF. Contributed by David Mollitor. 2019-10-12 13:26:47 +05:30
Ayush Saxena
eeb58a07e2 HDFS-14898. Use Relative URLS in Hadoop HDFS HTTP FS. Contributed by David Mollitor. 2019-10-10 01:03:33 +05:30
Surendra Singh Lilhore
d76e2655ac HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 2019-10-09 23:20:28 +05:30
Chen Liang
72ae371e7a HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-08 11:56:52 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Ayush Saxena
761594549e HDFS-14814. RBF: RouterQuotaUpdateService supports inherited rule. Contributed by Jinglun. 2019-10-08 14:01:44 +05:30
Surendra Singh Lilhore
382967be51 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-08 00:14:30 +05:30
Eric Yang
aa24add8f0 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle
 2019-10-04 13:13:10 -04:00
Takanobu Asanuma
b23bdaf085 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 2019-10-04 16:47:06 +09:00
Stephen O'Donnell
c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00
Virajith Jalaparti
844b766da5 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)" 2019-10-03 21:32:48 -07:00
Ayush Saxena
a3fe404244 HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C. 2019-10-03 22:34:55 +05:30
Ayush Saxena
5a7483ca5c HDFS-14888. RBF: Enable Parallel Test Profile for builds. Contributed by Ayush Saxena. 2019-10-03 19:21:27 +05:30
Chen Liang
1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Akira Ajisaka
3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li. 2019-10-02 11:03:16 +09:00
Wei-Chiu Chuang
6ef6594c7e HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
 2019-10-01 21:16:46 +05:30
Konstantin V Shvachko
b3275ab1f2 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:48:10 -07:00
Konstantin V Shvachko
760b523e58 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-29 13:11:31 -07:00
Inigo Goiri
d8313b2274 HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 2019-09-28 17:20:44 -07:00
Ayush Saxena
2200871607 HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun. 2019-09-28 22:03:53 +05:30
Ayush Saxena
c4c8d5fd0e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-28 21:44:44 +05:30
Ayush Saxena
0d5d0b914a Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao." 
This reverts commit ce58c05f1d.
 2019-09-28 21:32:15 +05:30
Sahil Takiar
13b427fc05 HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
 2019-09-27 12:25:03 -07:00
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Zhankun Tang
2870668cfa Make upstream aware of 3.1.3 release. 2019-09-27 16:44:07 +08:00
Chen Liang
2adcc3c932 HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:48:19 -07:00
Inigo Goiri
b1e55cfb55 HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He. 2019-09-26 15:26:23 -07:00
Ayush Saxena
1a2a352ecd HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya. 2019-09-26 20:45:37 +05:30
Ayush Saxena
a3f68933d7 HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui. 2019-09-26 19:54:07 +05:30
Takanobu Asanuma
bdaaa3bbf2 HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina 2019-09-25 23:48:49 +09:00
Steve Loughran
e346e3638c HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
 2019-09-25 12:16:41 +01:00
Prabhu Joseph
3f89084ac7
HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-09-25 10:19:30 +09:00
Ayush Saxena
66400c1cbb HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 2019-09-25 01:31:15 +05:30
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Inigo Goiri
afa1006a53 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor. 2019-09-24 09:58:42 -07:00
Inigo Goiri
816d3cb908 HDFS-14837. Review of Block.java. Contributed by David Mollitor. 2019-09-24 09:54:09 -07:00
Wei-Chiu Chuang
43203b466d HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-24 09:38:58 -07:00
Ayush Saxena
c30e495557 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-23 21:22:50 +05:30
Vinayakumar B
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494). Contributed by Vinayakumar B. 2019-09-23 12:37:47 +05:30
Rohith Sharma K S
4c0a7a9e13 Make upstream aware of 3.2.1 release. 2019-09-23 06:20:54 +00:00
Rakesh Radhakrishnan
659c88801d HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 2019-09-22 22:02:54 +05:30
Ayush Saxena
aa938662f9 HDFS-14833. RBF: Router Update Doesn't Sync Quota. Contributed by Ayush Saxena. 2019-09-21 09:53:13 +05:30
Inigo Goiri
3f223bebfa HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun. 2019-09-20 11:45:01 -07:00
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Inigo Goiri
a79f286c6f HDFS-14609. RBF: Security should use common AuthenticationFilter. Contributed by Chen Zhang. 2019-09-19 11:00:10 -07:00
Malcolm Taylor
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM. 
Contributed by Malcolm Taylor.

Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
 2019-09-19 16:00:05 +01:00
Chen Liang
4ed0aefe9f HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 16:47:46 -07:00
Wei-Chiu Chuang
5dd859a8a0 HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-09-18 09:34:01 -07:00
Sahil Takiar
3cf6e4272f HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-17 15:51:39 -07:00
Inigo Goiri
f580a87079 HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun. 2019-09-17 14:55:34 -07:00
Ayush Saxena
e04b8a46c3 HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina. 2019-09-14 10:39:20 +05:30
Stephen O'Donnell
06ad6540b3 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-13 10:36:45 -07:00
Surendra Singh Lilhore
4852a90e4b HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 2019-09-13 00:17:13 +05:30
Surendra Singh Lilhore
2ff2a7f612 HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina. 2019-09-12 19:43:18 +05:30
Surendra Singh Lilhore
d1c303a497 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 2019-09-12 19:11:50 +05:30
Akira Ajisaka
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor. 2019-09-12 12:41:57 +09:00
Takanobu Asanuma
c255333e20 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 2019-09-11 16:54:08 +09:00
Takanobu Asanuma
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414) 2019-09-11 10:32:07 +09:00
Erik Krogen
c92a3e94d8 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2019-09-06 10:22:28 -07:00
Surendra Singh Lilhore
d98c54816d HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green. 2019-09-06 18:12:42 +05:30
Surendra Singh Lilhore
fa7f03fc56 HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C. 2019-09-06 17:55:30 +05:30
Inigo Goiri
494d75eb2b HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang. 2019-09-05 21:15:17 -07:00
Inigo Goiri
72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Ayush Saxena
2f70b52a5b HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq. 2019-09-05 08:29:56 +05:30
Surendra Singh Lilhore
05704754a0 HDFS-14777. RBF: Set ReadOnly is failing for mount Table but actually readonly succeed to set. Contributed by Ranith Sardar. 2019-09-04 15:09:09 +05:30
Xiaoyu Yao
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 2019-09-03 09:29:58 -07:00
Ayush Saxena
91b01a1db7 HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena. 2019-09-03 18:08:42 +05:30
Surendra Singh Lilhore
5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
Stephen O'Donnell
915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-02 09:47:04 -07:00
Wei-Chiu Chuang
d207aba026 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit 7bebad61d9.
 2019-09-02 09:43:44 -07:00
Ayush Saxena
040f6e93bb HDFS-14654. RBF: TestRouterRpc#testNamenodeMetrics is flaky. Contributed by Chen Zhang. 2019-09-02 13:01:52 +05:30
Ayush Saxena
c3abfcefdd HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena. 2019-09-02 07:24:04 +05:30
Ayush Saxena
18d74fe41c HDFS-14711. RBF: RBFMetrics throws NullPointerException if stateStore disabled. Contributed by Chen Zhang. 2019-09-01 12:53:10 +05:30
Surendra Singh Lilhore
96f7dc1992 HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui. 2019-08-30 14:31:05 +05:30
Ayush Saxena
e220dac15c HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:40:13 +05:30
leosunli
915271245b Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-08-29 18:35:26 -07:00
Stephen O'Donnell
7bebad61d9 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 17:38:07 -07:00
Istvan Fajth
fcb7884bfc HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 16:56:14 -07:00
Inigo Goiri
3b22fcd377 HDFS-14256. Review Logging of NameNode Class. Contributed by David Mollitor. 2019-08-29 14:21:09 -07:00
Inigo Goiri
ffca734c62 HDFS-14104. Review getImageTxIdToRetain. Contributed by David Mollitor. 2019-08-29 14:18:35 -07:00
He Xiaoqiao
f600fbb6c4 HDFS-11246. FSNameSystem#logAuditEvent should be called outside the read or write locks. Contributed by He Xiaoqiao, Kuhu Shukla. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Kuhu Shukla <kshukla@apache.org>
 2019-08-29 10:10:52 -07:00
Ayush Saxena
8e779a151e HDFS-14721. RBF: ProxyOpComplete is not accurate in FederationRPCPerformanceMonitor. Contributed by xuzq. 2019-08-29 20:08:38 +05:30
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
Inigo Goiri
48cb583906 HDFS-14710. RBF: Improve some RPC performance by using previous block. Contributed by xuzq. 2019-08-28 10:48:00 -07:00
CR Hota
6e37d65b03 HDFS-14760. Log INFO mode if snapshot usage and actual usage differ. Contributed by CR Hota. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 16:20:25 -07:00
He Xiaoqiao
dde9399b37 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 15:26:59 -07:00
Shashikant Banerjee
ce8eb1283a Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee." 
This reverts commit 62445021d5 as it has unintended changes in DirectoryWithSnapshotFeature class..
 2019-08-27 23:23:44 +05:30
Jonathan Hung
8ab7020e64 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 2019-08-27 10:30:31 -07:00
Takanobu Asanuma
b69ac575a1 HDFS-14772. RBF: hdfs-rbf-site.xml can't be loaded automatically. Contributed by Yuxuan Wang 2019-08-27 14:32:32 +09:00
Arpit Agarwal
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 2019-08-26 15:57:55 -07:00
Ayush Saxena
d2225c8ca8 HDFS-14722. RBF: GetMountPointStatus should return mountTable information when getFileInfoAll throw IOException. Contributed by xuzq. 2019-08-24 08:47:43 +05:30
Inigo Goiri
0b796754b9 HDFS-14766. RBF: MountTableStoreImpl#getMountTableEntries returns extra entry. Contributed by Chen Zhang. 2019-08-23 14:53:57 -07:00
Chen Liang
ebef99dcf4 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:18:43 -07:00
Erik Krogen
d699022fce HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 2019-08-23 11:33:21 -07:00
Wang Yuxuan
894e2300d6 HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly (#1334) 
HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly
 2019-08-23 10:46:39 -07:00
Ayush Saxena
c92de8209d HDFS-14747. RBF: IsFileClosed should be return false when the file is open in multiple destination. Contributed by xuzq. 2019-08-23 19:08:07 +05:30
Akira Ajisaka
bd7baea5a5
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 2019-08-23 15:28:27 +09:00
Akira Ajisaka
abc8fde4ca
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 2019-08-23 14:44:34 +09:00
Stephen O'Donnell
b67812ea21 HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index (#1028). Contributed by Stephen O'Donnell. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-08-22 17:09:57 -07:00
Stephen O'Donnell
93daf69f90 HDFS-14675. Increase Balancer Defaults Further. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Gabor Bota <gabota@apache.org>
 2019-08-22 15:21:33 -07:00
Inigo Goiri
e04dcfdc57 HDFS-14583. FileStatus#toString() will throw IllegalArgumentException. Contributed by xuzq. 2019-08-22 10:22:38 -07:00
Surendra Singh Lilhore
76790a1e67 HDFS-14358. Provide LiveNode and DeadNode filter in DataNode UI. Contributed by hemanthboyina. 2019-08-22 12:27:45 +05:30
Ayush Saxena
52c77bc160 HDFS-14741. RBF: RecoverLease should be return false when the file is open in multiple destination. Contributed by xuzq 2019-08-22 08:57:22 +05:30
Ayush Saxena
f9029c4070 HDFS-14744. RBF: Non secured routers should not log in error mode when UGI is default. Contributed by CR Hota. 2019-08-22 02:40:39 +05:30
Ayush Saxena
0f598aed13 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 02:25:06 +05:30
Inigo Goiri
93595febaa HDFS-14756. RBF: getQuotaUsage may ignore some folders. Contributed by Chen Zhang. 2019-08-21 09:39:57 -07:00
Inigo Goiri
5eeb6da2d4 HDFS-14714. RBF: implement getReplicatedBlockStats interface. Contributed by Chen Zhang. 2019-08-21 09:38:17 -07:00
Wei-Chiu Chuang
57f7370174 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit b58a35f374.
 2019-08-21 08:59:49 -07:00
Anu Engineer
8aaf5e1a14 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.
 2019-08-20 20:24:19 -07:00
Surendra Singh Lilhore
3a145e2918 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-20 15:55:26 -07:00
Sandeep Kumar
269b543367 HADOOP-16523. Minor spell mistake in comment : Line number 101 (#388) 2019-08-20 13:36:23 -07:00
Wei-Chiu Chuang
4cb22cd867 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:37:11 -07:00
Wei-Chiu Chuang
aa6995fde2 HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 2019-08-20 10:32:55 -07:00
Sunil G
bd92462321 HDFS-14729. Upgrade Bootstrap and jQuery versions used in HDFS UIs. Contributed by Vivek Ratnavel Subramanian. This closes #1297 2019-08-20 18:42:19 +05:30
Shashikant Banerjee
62445021d5 HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee. 2019-08-20 14:37:01 +05:30
Wei-Chiu Chuang
360a96f342 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 2019-08-19 13:08:55 -07:00
HUAN-PING SU
abae6ff2a2 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-19 11:14:51 -07:00
Surendra Singh Lilhore
b8db5b9a98 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 18:13:35 -07:00
Wei-Chiu Chuang
b58a35f374 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:46:00 -07:00
Wei-Chiu Chuang
d873ddd656 HDFS-14127. Add a description about the observer read configuration. Contributed by xiangheng. 2019-08-17 04:39:22 -07:00
Wei-Chiu Chuang
528378784f HDFS-12012. Fix spelling mistakes in BPServiceActor.java. Contributed by chencan. 2019-08-17 04:37:05 -07:00
hunshenshi
a38b9e137e HDFS-14456:HAState#prepareToEnterState neednt a lock (#770) Contributed by hunshenshi. 2019-08-16 14:53:06 -07:00
LeonGao91
9a1d8cfaf5 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 2019-08-16 08:00:51 -07:00
Surendra Singh Lilhore
34681643e9 HDFS-14719. Correct the safemode threshold value in BlockManagerSafeMode. Contributed by hemanthboyina. 2019-08-15 12:12:18 +05:30
Shashikant Banerjee
0a85af959c HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 2019-08-15 10:16:25 +05:30
Ayush Saxena
b06c2345ef HDFS-14713. RBF: RouterAdmin supports refreshRouterArgs command but not on display. Contributed by wangzhaohui. 2019-08-14 20:44:31 +05:30
Siyao Meng
3c0382f1b9 HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2019-08-14 07:30:35 -07:00
Wei-Chiu Chuang
c13ec7ab66 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 2019-08-13 17:15:26 -07:00
Wei-Chiu Chuang
633b7c1cfe HDFS-14625. Make DefaultAuditLogger class in FSnamesystem to Abstract. Contributed by hemanthboyina. 2019-08-13 16:50:49 -07:00
Masatake Iwasaki
da0006fe04 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-14 08:39:40 +09:00
Siyao Meng
6ae8bc3a4a HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng. 2019-08-13 16:27:57 -07:00
Ayush Saxena
e9b6b81de4 HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng. 2019-08-13 19:17:10 +05:30
Ayush Saxena
454420e4f2 HDFS-14708. TestLargeBlockReport#testBlockReportSucceedsWithLargerLengthLimit fails in trunk. Contributed by Lisheng Sun. 2019-08-13 08:27:20 +05:30
Siyao Meng
c92b49876a HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 2019-08-12 17:26:20 -07:00
Yiqun Lin
8a77a224c7 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-09 18:40:28 -07:00
Siyao Meng
e02ffed1b1 HDFS-12125. Document the missing EC removePolicy command (#1258) Contributed by Wenxin He, Siyao Meng. 
Co-authored-by: Wenxin He <wenxin.he@gmail.com>
 2019-08-09 18:00:22 -07:00
Wei-Chiu Chuang
ce3c5a3e3b HDFS-14623. In NameNode Web UI, for Head the file (first 32K) old data is showing. Contributed by hemanthboyina. 2019-08-09 15:41:37 -07:00
Wei-Chiu Chuang
865021b8c9 HDFS-14195. OIV: print out storage policy id in oiv Delimited output. Contributed by Wang, Xinglong. 2019-08-09 15:37:29 -07:00
Rakesh Radhakrishnan
f6fa865d6f HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 2019-08-09 14:07:54 +05:30
Wei-Chiu Chuang
6ad9a11494 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 2019-08-08 13:50:30 -07:00
Wei-Chiu Chuang
2265872c2d HDFS-14705. Remove unused configuration dfs.min.replication. Contributed by CR Hota. 2019-08-08 13:48:29 -07:00
Wei-Chiu Chuang
28a848412c HDFS-14701. Change Log Level to warn in SlotReleaser. Contributed by Lisheng Sun. 2019-08-08 13:46:31 -07:00
Wei-Chiu Chuang
23f91f68b8 HDFS-14662. Document the usage of the new Balancer "asService" parameter. Contributed by Chen Zhang. 2019-08-08 13:45:29 -07:00
Stephen O'Donnell
b0799148cf HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-08 13:36:39 -07:00
Ayush Saxena
11f750e6a7 HDFS-14616. Add the warn log when the volume available space isn't enough. Contributed by liying. 2019-08-08 00:49:58 +05:30
Ayush Saxena
de64253892 HDFS-14608. DataNode#DataTransfer should be named. Contributed by Inigo Goiri. 2019-08-08 00:37:56 +05:30
Erik Krogen
827dbb11e2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 2019-08-07 09:25:58 -07:00
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Lokesh Jain
954ff36360 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 15:30:50 -07:00
Stephen O'Donnell
35e0a01d7b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 07:57:46 -07:00
Ayush Saxena
11272159bb HDFS-14679. Failed to add erasure code policies with example template. Contributed by Yuan Zhou. 2019-08-06 08:42:15 +05:30
Ayush Saxena
8f40856f76 HDFS-14669. TestDirectoryScanner#testDirectoryScannerInFederatedCluster fails intermittently in trunk. Contributed by qiang Liu. 2019-08-03 12:15:41 +05:30
Ayush Saxena
e503db5f44 HDFS-12826. Document Saying the RPC port, But it's required IPC port in HDFS Federation Document. Contributed by usharani. 2019-08-03 12:05:46 +05:30
Erik Krogen
e7a0b8aa83 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 2019-08-02 11:48:31 -07:00
Wei-Chiu Chuang
b964b81f85 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 2019-08-02 10:54:32 -07:00
Wei-Chiu Chuang
c2d00c8450 HDFS-13131. Modifying testcase testEnableAndDisableErasureCodingPolicy. Contributed by chencan. 2019-08-01 19:11:43 -07:00
Siyao Meng
17e8cf501b HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 2019-08-01 17:15:22 -07:00
Siyao Meng
99bf1dc9eb HDFS-14683. WebHDFS: Add erasureCodingPolicy field to GETCONTENTSUMMARY response (#1189) Contributed by Siyao Meng. 2019-08-01 17:14:07 -07:00
Wei-Chiu Chuang
32607dbd98 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-01 15:48:36 -07:00
Ayush Saxena
89b102f916 HDFS-14661. RBF: updateMountTableEntry shouldn't update mountTableEntry if targetPath not exist. Contributed by xuzq. 2019-08-01 08:43:39 +05:30
Chao Sun
b008072044 HDFS-14681. RBF: TestDisableRouterQuota failed because port 8888 was occupied. 2019-07-31 10:24:48 -07:00
Wei-Chiu Chuang
99f88c30cb HDFS-14419. Avoid repeated calls to the listOpenFiles function. Contributed by HuangTao. 2019-07-30 16:50:06 -07:00
Chao Sun
3ae775d740 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 16:01:17 -07:00
Erik Krogen
1f26cc8705 HDFS-13783. Add an option to the Balancer to make it run as a long-running service. Contributed by Chen Zhang. 2019-07-30 15:42:55 -07:00
Inigo Goiri
7849bdcf70 HDFS-14449. Expose total number of DT in JMX for Namenode. Contributed by Fengnan Li. 2019-07-30 13:45:27 -07:00
Ayush Saxena
8deced0073 HDFS-14677. TestDataNodeHotSwapVolumes#testAddVolumesConcurrently fails intermittently in trunk. Contributed by Chen Zhang. 2019-07-30 22:17:39 +05:30
Wei-Chiu Chuang
8053085388 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:31:34 -07:00
CR Hota
611718f59f HDFS-14670: RBF: Create secret manager instance using FederationUtil#newInstance. 2019-07-29 11:00:22 -07:00
Ayush Saxena
02bd02b5af HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-07-28 08:41:42 +05:30
hunshenshi
2fe450cb5e HDFS-14425. Native build fails on macos due to jlong in hdfs.c (#741) 2019-07-26 15:33:12 -07:00
HUAN-PING SU
ecc8acfd24 HDFS-14673. The console log is noisy when using DNSDomainNameResolver to resolve NameNode. 2019-07-26 11:08:46 -07:00
Ayush Saxena
ce99cc31e9 HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-07-26 08:31:00 +05:30
Masatake Iwasaki
6b8107ad97 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-07-26 09:35:23 +09:00
Ayush Saxena
62deab17a3 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 2019-07-25 06:51:07 +05:30
Ayush Saxena
377f95bbe8 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 2019-07-23 08:37:55 +05:30
Ayush Saxena
6282c02d7d HDFS-14577. RBF: FederationUtil#newInstance should allow constructor without context. Contributed by CR Hota. 2019-07-19 17:44:13 +05:30
Takanobu Asanuma
9792f58662 HDFS-14653. RBF: Correct the default value for dfs.federation.router.namenode.heartbeat.enable. Contributed by Ayush Saxena. 2019-07-18 10:16:20 +09:00
Chen Liang
5e6cc6fe8a HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-07-17 14:37:16 -07:00
Ayush Saxena
af0665c7f5 HDFS-13647. Fix the description of storageType option for space quota. Contributed by Takanobu Asanuma. 2019-07-17 23:22:32 +05:30
Ayush Saxena
f77d54c243 HDFS-14642. processMisReplicatedBlocks does not return correct processed count. Contributed by Stephen O'Donnell. 2019-07-16 08:14:27 +05:30
Ayush Saxena
64d4abf489 HDFS-14593. RBF: Implement deletion feature for expired records in State Store. Contributed by Takanobu Asanuma. 2019-07-15 22:38:00 +05:30
Rakesh Radhakrishnan
30a8f840f1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 2019-07-15 13:18:23 +05:30
Rakesh Radhakrishnan
e98adb00b7 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 2019-07-15 13:02:37 +05:30
Shashikant Banerjee
f9fab9f22a HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 2019-07-12 15:42:06 +05:30
Masatake Iwasaki
00dd843a1a
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-12 10:04:55 +09:00
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Inigo Goiri
5747f6cff5 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 2019-07-11 08:55:44 -07:00
Inigo Goiri
eccc9a40de HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-10 11:11:52 -07:00
Ajay Yadav
cdb20adfcc HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. (#973) 2019-07-10 11:03:58 -07:00
Yiqun Lin
993dc8726b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 2019-07-10 10:53:34 +08:00
Erik Krogen
4632708148 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-08 14:46:25 -07:00
Wei-Chiu Chuang
ec851e4db2 HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 2019-07-08 15:23:46 +08:00
leosunli
9c90729486 HDFS-13694. Making md5 computing being in parallel with image loading. 2019-07-05 10:17:37 -07:00
Inigo Goiri
c3ca348b81 HDFS-14620. RBF: Fix 'not a super user' error when disabling a namespace in kerberos with superuser principal. Contributed by luhuachao. 2019-07-04 11:21:55 -07:00
Masatake Iwasaki
1c254a8ec7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 2019-07-04 23:20:42 +09:00
Weiwei Yang
729cb3aefe HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-04 09:37:24 +08:00
paulward24
d8bac50e12 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 2019-07-01 13:54:49 -07:00
paulward24
d203045c30 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 2019-06-28 16:06:43 -07:00
Inigo Goiri
4a212242d9 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 2019-06-27 10:00:05 -07:00
Inigo Goiri
041e0c0564 HDFS-14036. RBF: Add hdfs-rbf-default.xml to HdfsConfiguration by default. Contributed by Takanobu Asanuma. 2019-06-27 09:57:52 -07:00
Anu Engineer
041e7a7dee HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.
 2019-06-24 19:07:22 -07:00
Inigo Goiri
b76b843c8b HDFS-13371. NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X. Contributed by Sherwood Zheng. 2019-06-24 17:52:33 -07:00
Inigo Goiri
38a560c6f1
Merge pull request #977 from leosunli/trunk 
HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
 2019-06-24 17:44:54 -07:00
Ajay Kumar
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Inigo Goiri
719d57bf46 HDFS-14545. RBF: Router should support GetUserMappingsProtocol. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Akira Ajisaka
8627f6532b HDFS-14550. RBF: Failed to get statistics from NameNodes before 2.9.0. Contributed by He Xiaoqiao. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ade8d3b60e HDFS-14508. RBF: Clean-up and refactor UI components. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena
6915d7e13c HDFS-13480. RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ddbe08db33 HDFS-13955. RBF: Support secure Namenode in NamenodeHeartbeatService. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Inigo Goiri
b062dd462d HDFS-14475. RBF: Expose router security enabled status on the UI. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena
b6fff8c81e HDFS-13787. RBF: Add Snapshot related ClientProtocol APIs. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena
d240eec136 HDFS-14516. RBF: Create hdfs-rbf-site.xml for RBF specific properties. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena
9b197c2893 HDFS-13909. RBF: Add Cache pools and directives related ClientProtocol APIs. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena
ffbb6b6557 HDFS-13255. RBF: Fail when try to remove mount point paths. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena
8e4267650f HDFS-14440. RBF: Optimize the file write process in case of multiple destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula
2636a54ffd HDFS-13995. RBF: Security documentation. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola
393f15176d HDFS-14447. RBF: Router should support RefreshUserMappingsProtocol. Contributed by Shen Yinjie. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola
3deb5d345f HDFS-14490. RBF: Remove unnecessary quota checks. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena
62fa53a01d HDFS-14210. RBF: ACL commands should work over all the destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola
32841178ba HDFS-14426. RBF: Add delegation token total count as one of the federation metrics. Contributed by Fengnan Li. 2019-06-24 22:03:04 +05:30
Ayush Saxena
203664e6b2 HDFS-14454. RBF: getContentSummary() should allow non-existing folders. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena
b522b52bb1 HDFS-14457. RBF: Add order text SPACE in CLI command 'hdfs dfsrouteradmin'. Contributed by luhuachao. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula
021a43b1a4 HDFS-13972. RBF: Support for Delegation Token (WebHDFS). Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena
de7da9b69e HDFS-14422. RBF: Router shouldn't allow READ operations in safe mode. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena
0f9b8d7a75 HDFS-14369. RBF: Fix trailing / for webhdfs. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena
dd8c2b92df HDFS-13853. RBF: RouterAdmin update cmd is overwriting the entry not updating the existing. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena
6c42d40504 HDFS-14316. RBF: Support unavailable subclusters for mount points with multiple destinations. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena
0dbd87874a HDFS-14388. RBF: Prevent loading metric system when disabled. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Ayush Saxena
9a9fbbe145 HDFS-14351. RBF: Optimize configuration item resolving for monitor namenode. Contributed by He Xiaoqiao and Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola
f539e2a4ee HDFS-14343. RBF: Fix renaming folders spread across multiple subclusters. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola
2a2d5eb441 HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri
fcabc8f0e4 HDFS-14335. RBF: Fix heartbeat typos in the Router. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri
6cdf8db55c HDFS-14331. RBF: IOE While Removing Mount Entry. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri
1ce25e702b HDFS-14329. RBF: Add maintenance nodes to federation metrics. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri
7bbe01a196 HDFS-14259. RBF: Fix safemode message for Router. Contributed by Ranith Sadar. 2019-06-24 22:03:03 +05:30
Inigo Goiri
1c7ab59be3 HDFS-14322. RBF: Security manager should not load if security is disabled. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula
c4b1fa91fa HDFS-14052. RBF: Use Router keytab for WebHDFS. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri
de719b08b5 HDFS-14307. RBF: Update tests to use internal Whitebox instead of Mockito. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola
8b8ff5ccbc HDFS-14249. RBF: Tooling to identify the subcluster location of a file. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola
50aee18a84 HDFS-14268. RBF: Fix the location of the DNs in getDatanodeReport(). Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri
e2a3c4494b HDFS-14226. RBF: Setting attributes should set on all subclusters' directories. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula
75f8b6ccfa HDFS-13358. RBF: Support for Delegation Token (RPC). Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri
7e63e37dc5 HDFS-14230. RBF: Throw RetriableException instead of IOException when no namenodes available. Contributed by Fei Hui. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola
912b90f91e HDFS-14252. RBF : Exceptions are exposing the actual sub cluster path. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore
9c4e55685b HDFS-14225. RBF : MiniRouterDFSCluster should configure the failover proxy provider for namespace. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Inigo Goiri
9eed3a49df HDFS-14215. RBF: Remove dependency on availability of default namespace. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30