big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,373 Commits 2 Branches 2 Tags 264 MiB
1385cdfeb9
Commit Graph

5189 Commits

Author SHA1 Message Date
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Ayush Saxena
b735a77717 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:06:16 +05:30
Ayush Saxena
8525874561 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:53:33 +05:30
Uma Maheswara Rao G
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 2020-06-06 08:11:57 -07:00
Ayush Saxena
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 2020-06-05 22:19:38 -07:00
Ayush Saxena
2326123705 HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena. 2020-06-05 19:15:05 +05:30
Toshihiro Suzuki
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.
 2020-06-05 11:11:49 +01:00
Ayush Saxena
e8cb2ae409 HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 2020-06-03 12:47:15 +05:30
Ayush Saxena
f6453244ab HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan. 2020-06-03 12:16:36 +05:30
Uma Maheswara Rao G
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Ayush Saxena
a838d871a7 HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2020-05-28 20:52:58 +05:30
Inigo Goiri
2148a8fe64 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 2020-05-27 11:06:13 -07:00
Ayush Saxena
f4901d0778 HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun. 2020-05-24 17:59:16 +05:30
Inigo Goiri
6e04b00df1 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 2020-05-23 09:58:19 -07:00
Ayush Saxena
e0ae232f66 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 2020-05-23 18:38:32 +05:30
Ayush Saxena
f99fcb26ab HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2020-05-23 18:29:31 +05:30
Takanobu Asanuma
4d22d1c58f HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 2020-05-23 17:28:38 +09:00
Uma Maheswara Rao G
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 2020-05-21 21:34:58 -07:00
leosunli
be374faf42
HDFS-13639. SlotReleaser is not fast enough (#1885) 2020-05-21 13:21:17 -07:00
Abhishek Das
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 2020-05-18 22:27:12 -07:00
Chen Liang
7bb902bc0d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 10:58:52 -07:00
Wei-Chiu Chuang
2abcf7762a HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang
4525292d41 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit 86e6aa8eec.
 2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang
50caba1a92 HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 2020-05-18 08:40:38 -07:00
He Xiaoqiao
a3f44dacc1 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-18 07:09:43 -07:00
pustota2009
86e6aa8eec
HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
 2020-05-18 07:04:04 -07:00
Uma Maheswara Rao G
ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 2020-05-14 17:29:35 -07:00
Surendra Singh Lilhore
743c2e9071 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 2020-05-13 15:01:07 +05:30
S O'Donnell
29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Takanobu Asanuma
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. (#2008) 2020-05-12 09:04:03 -05:00
Ayush Saxena
0fe49036e5 HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu. 2020-05-12 13:11:31 +05:30
Ayush Saxena
4c53fb9ce1 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:48:34 +05:30
Ayush Saxena
cb64e993c2 HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi. 2020-05-10 00:11:46 +05:30
Ayush Saxena
d59de27c01 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:53:49 +05:30
Inigo Goiri
130f89e068 HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 2020-05-06 13:18:32 -07:00
Ayush Saxena
7fddf4855e HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 16:55:04 +05:30
Uma Maheswara Rao G
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 2020-05-04 17:55:40 -07:00
Konstantin V Shvachko
ebb878bab9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 2020-05-04 10:29:50 -07:00
Ayush Saxena
d6fc482a54 Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:09:14 +05:30
Mingliang Liu
44de193bec
HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li 2020-05-02 20:56:29 -07:00
Akira Ajisaka
f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:02:27 -07:00
Shashikant Banerjee
82343790ee HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-05-01 12:16:02 +05:30
S O'Donnell
9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
Akira Ajisaka
5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 2020-04-28 06:17:14 +09:00
Mingliang Liu
74d4a18735
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:43:02 -07:00
Dhiraj
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:04:32 -07:00
Mingliang Liu
5958af4dbc
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:31:02 -07:00
Toshihiro Suzuki
1824aee9da
HDFS-15217 Add more information to longest write/read lock held log 2020-04-18 13:52:07 -07:00
Ayush Saxena
37d6582223 HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 2020-04-17 13:32:02 +05:30
Stephen O'Donnell
3481895f8a HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-04-16 20:18:25 -07:00
Wei-Chiu Chuang
061afcdf30
HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-04-09 09:19:35 -07:00
He Xiaoqiao
d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-01 16:36:46 -07:00
Ayush Saxena
960c9ebaea HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 2020-03-30 02:40:25 +05:30
Ayush Saxena
696a663cd0 HDFS-15193. Improving the error message for missing dfs.namenode.rpc-address.$NAMESERVICE. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Chen Liang
f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00
Inigo Goiri
cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Arpit Agarwal
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. 
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
 2020-03-25 10:28:30 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Ayush Saxena
ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Karthik Palanisamy
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
He Xiaoqiao
af64ce2f4a HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 12:43:51 -07:00
Takanobu Asanuma
b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
Ayush Saxena
f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena
bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Wei-Chiu Chuang
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
 2020-03-18 13:27:13 +00:00
Ayush Saxena
8a7c54995a HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina. 2020-03-15 16:16:27 +05:30
He Xiaoqiao
e9955bb8ff HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829) 
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
 2020-03-13 11:29:44 -07:00
Ayush Saxena
fb1d7635ae HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang. 2020-03-13 21:10:03 +05:30
Wei-Chiu Chuang
20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface 
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
 2020-03-09 14:51:16 +00:00
Inigo Goiri
4062217189 HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena. 2020-03-06 10:14:53 -08:00
Inigo Goiri
3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Inigo Goiri
1d4d0fcbe1 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 2020-03-04 10:02:54 -08:00
Ayush Saxena
1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Inigo Goiri
97b797c314 HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun. 2020-02-28 18:47:22 -08:00
Ayush Saxena
429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Surendra Singh Lilhore
2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena
209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-25 13:28:50 -08:00
Ayush Saxena
7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena
ba9025c7cd HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun. 2020-02-22 14:27:26 +05:30
Konstantin V Shvachko
ecbcb058b8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 2020-02-21 17:53:37 -08:00
Konstantin V Shvachko
6f84269bcd HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 2020-02-21 10:51:14 -08:00
Masatake Iwasaki
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 2020-02-21 11:56:07 +09:00
Inigo Goiri
ec7507162c HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham. 2020-02-19 11:33:58 -08:00
Ayush Saxena
ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Ayush Saxena
84f7638840 HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri. 2020-02-18 00:43:33 +05:30
Surendra Singh Lilhore
810783d443 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-02-16 12:15:03 +05:30
Ayush Saxena
c75756fe13 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore
a98352ced1 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 16:57:41 +05:30
Kihwal Lee
9b8a78d97b HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.
 2020-02-11 12:40:00 -06:00
Stephen O'Donnell
d7c136b9ed HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-11 08:00:15 -08:00
Ayush Saxena
6191d4b4a0 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 2020-02-09 23:32:22 +05:30
Ayush Saxena
d23317b102 HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:33:57 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Chen Liang
ce7b8b5634 HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:12:35 -08:00
Kihwal Lee
799d4c1cf4 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.
 2020-01-29 11:00:27 -06:00
Chen Liang
483397c7f7 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-28 15:20:36 -08:00
Chen Liang
3e86807802 Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang." 
This reverts commit ff8ff0f7e5.
 2020-01-28 15:19:47 -08:00
Chen Liang
ff8ff0f7e5 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang. 2020-01-28 15:14:58 -08:00
Inigo Goiri
1839c467f6 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 2020-01-28 10:10:35 -08:00
Ayush Saxena
87c198468b HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:09:38 +05:30
Ayush Saxena
f876dc228b HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena. 2020-01-28 19:34:21 +05:30
Ayush Saxena
6d008c0d39 HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest 2020-01-25 11:47:29 +05:30
Kihwal Lee
d10f77e3c9 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.
 2020-01-24 09:15:27 -06:00
Ayush Saxena
92c58901d7 HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena 2020-01-23 18:18:34 +05:30
Inigo Goiri
8cfc3673dc HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 2020-01-21 13:29:20 -08:00
Inigo Goiri
b657822b98 HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein. 2020-01-21 13:22:53 -08:00
Chao Sun
d7c4f8ab21 HDFS-13616. Batch listing of multiple directories (#1725) 2020-01-15 17:22:47 -08:00
Ayush Saxena
cebce0a348 HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:29:11 +05:30
Chen Liang
26a969ec73 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 2020-01-10 17:06:59 -08:00
Kihwal Lee
5fb901ac40 HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 2020-01-10 12:43:21 -06:00
Surendra Singh Lilhore
bf45f3b80a HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:04:05 +05:30
Masatake Iwasaki
aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 2020-01-08 16:45:39 +09:00
Masatake Iwasaki
a43c177f1d HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 2020-01-08 11:28:34 +09:00
Masatake Iwasaki
037ec8cfb1 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-01-04 01:55:36 +09:00
Rakesh Radhakrishnan
d79cce20ab HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 2020-01-02 11:44:00 +05:30
Ayush Saxena
300505c562 HDFS-15054. Delete Snapshot not updating new modification time. Contributed by hemanthboyina. 2019-12-25 12:42:50 +05:30
Konstantin V Shvachko
b98ac2a3af HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 2019-12-24 12:08:34 -08:00
Inigo Goiri
b86895485d HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 2019-12-19 09:34:43 -08:00
Shashikant Banerjee
fdd96e46d1 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 2019-12-18 22:50:46 +05:30
Inigo Goiri
24080666e5 HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun. 2019-12-16 18:41:45 -08:00
Ayush Saxena
7a87007545 HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina. 2019-12-15 00:33:32 +05:30
Chen Liang
65c4660bcd HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 2019-12-12 10:22:05 -08:00
Erik Krogen
c174d50b30 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 2019-12-11 09:39:37 -08:00
Stephen O'Donnell
c93cb6790e HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-10 17:16:43 -08:00
Masatake Iwasaki
11cd5b6e39 HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-12-07 08:46:45 +09:00
Ayush Saxena
83a14559e5 HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui. 2019-12-05 18:52:01 +05:30
Ayush Saxena
0c217feed8 HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun. 2019-12-03 09:18:09 +05:30
Ayush Saxena
6b2d6d4aaf HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina. 2019-11-29 23:55:37 +05:30
Surendra Singh Lilhore
0384687811 HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-28 23:49:35 +05:30
Ayush Saxena
46166bd8d1 HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena. 2019-11-28 18:28:20 +05:30
Yiqun Lin
c3659f8f94 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. 2019-11-28 14:41:49 +08:00
Yiqun Lin
2b452b4e60 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 2019-11-28 10:43:35 +08:00
Yiqun Lin
c8bef4d6a6 HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. 2019-11-27 10:57:20 +08:00
Ayush Saxena
049940e77b HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:30:02 +05:30
Takanobu Asanuma
b25e94ce29 HDFS-14924. RenameSnapshot not updating new modification time. Contributed by hemanthboyina 2019-11-22 13:48:02 +09:00
Yiqun Lin
9b6906fe91 HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun. 2019-11-22 10:53:55 +08:00
Surendra Singh Lilhore
26270196a2 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:31:25 +05:30
He Xiaoqiao
0b50aa29fd HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
 2019-11-19 12:16:57 -08:00
Surendra Singh Lilhore
c870514740 HDFS-14992. TestOfflineEditsViewer is failing in Trunk. Contributed by hemanthboyina. 2019-11-20 00:07:43 +05:30
Ayush Saxena
34cb595d2e HDFS-14967. TestWebHDFS fails in Windows. Contributed by Renukaprasad C. 2019-11-18 18:49:58 +05:30
Yiqun Lin
b3119b9ab6 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. 2019-11-16 11:32:41 +08:00
Hui Fei
67f2c491fe HDFS-14802. The feature of protect directories should be used in RenameOp (#1669) 2019-11-15 12:35:37 -08:00
He Xiaoqiao
c892a879dd HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
 2019-11-15 12:16:31 -08:00
Erik Krogen
b2cc8b6b4a HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 2019-11-15 10:10:37 -08:00
Erik Krogen
586defe711 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 2019-11-13 11:46:35 -08:00
Inigo Goiri
40150da1e1 HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina. 2019-11-12 10:53:54 -08:00
Ayush Saxena
247584eb63 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:49:01 +05:30
Chen Liang
dd900259c4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 09:56:19 -08:00
Chen Liang
9d0d580031 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 09:30:04 -08:00
Surendra Singh Lilhore
c36014165c HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 2019-11-06 19:28:55 +05:30
Inigo Goiri
bfb8f28cc9 HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang. 2019-11-05 09:19:52 -08:00
Ayush Saxena
b643a1cbe8 HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun. 2019-11-05 09:23:20 +05:30
Ayush Saxena
2ffec347eb HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-05 01:37:15 +05:30
Zhao Junwang
de6b8b0c0b HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
 2019-11-01 16:37:53 -07:00
Ayush Saxena
7d7acb004a HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:50:10 +05:30
Surendra Singh Lilhore
02009c3bb7 HDFS-14768. EC : Busy DN replica should be consider in live replica check. Contributed by guojh. 2019-11-01 22:15:05 +05:30
Ayush Saxena
9d25ae7669 HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:49:37 +05:30
Ayush Saxena
d9fbedc4ae HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun. 2019-10-31 23:39:27 +05:30
Eric Yang
30ed24a421 HDFS-14730. Removed unused configuration dfs.web.authentication.filter. 
Contributed by Chen Zhang
 2019-10-28 19:27:32 -04:00
zhaoyim
30db895b59 HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-25 13:09:14 -07:00
Vinayakumar B
ee699dc26c HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:14:09 +05:30
Mukul Kumar Singh
a901405ad8 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-22 13:31:15 -07:00
Ayush Saxena
19f35cfd57 HDFS-14918. Remove useless getRedundancyThread from BlockManagerTestUtil. Contributed by Fei Hui. 2019-10-22 18:34:02 +05:30
Wei-Chiu Chuang
72003b19bf HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-21 17:31:43 -07:00
Ayush Saxena
74c2329fc3 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:35:12 +05:30
Ayush Saxena
447f46d962 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:10:59 +05:30
Surendra Singh Lilhore
54dc6b7d72 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 2019-10-17 22:28:47 +05:30
Ayush Saxena
5f4641a120 HDFS-14238. A log in NNThroughputBenchmark should change log level to INFO instead of ERROR. Contributed by Shen Yinjie. 2019-10-12 21:06:23 +05:30
Surendra Singh Lilhore
d76e2655ac HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 2019-10-09 23:20:28 +05:30
Chen Liang
72ae371e7a HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-08 11:56:52 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Surendra Singh Lilhore
382967be51 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-08 00:14:30 +05:30
Stephen O'Donnell
c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00
Virajith Jalaparti
844b766da5 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)" 2019-10-03 21:32:48 -07:00
Chen Liang
1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Wei-Chiu Chuang
6ef6594c7e HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
 2019-10-01 21:16:46 +05:30
Konstantin V Shvachko
b3275ab1f2 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:48:10 -07:00
Konstantin V Shvachko
760b523e58 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-29 13:11:31 -07:00
Ayush Saxena
2200871607 HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun. 2019-09-28 22:03:53 +05:30
Ayush Saxena
c4c8d5fd0e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-28 21:44:44 +05:30
Ayush Saxena
0d5d0b914a Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao." 
This reverts commit ce58c05f1d.
 2019-09-28 21:32:15 +05:30
Sahil Takiar
13b427fc05 HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
 2019-09-27 12:25:03 -07:00
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Ayush Saxena
c30e495557 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-23 21:22:50 +05:30
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Inigo Goiri
f580a87079 HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun. 2019-09-17 14:55:34 -07:00
Surendra Singh Lilhore
4852a90e4b HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 2019-09-13 00:17:13 +05:30
Surendra Singh Lilhore
d1c303a497 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 2019-09-12 19:11:50 +05:30
Erik Krogen
c92a3e94d8 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2019-09-06 10:22:28 -07:00
Inigo Goiri
494d75eb2b HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang. 2019-09-05 21:15:17 -07:00
Xiaoyu Yao
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 2019-09-03 09:29:58 -07:00
Ayush Saxena
91b01a1db7 HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena. 2019-09-03 18:08:42 +05:30
Stephen O'Donnell
915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-02 09:47:04 -07:00
Wei-Chiu Chuang
d207aba026 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit 7bebad61d9.
 2019-09-02 09:43:44 -07:00
leosunli
915271245b Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-08-29 18:35:26 -07:00
Stephen O'Donnell
7bebad61d9 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 17:38:07 -07:00
Istvan Fajth
fcb7884bfc HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 16:56:14 -07:00
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
Jonathan Hung
8ab7020e64 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 2019-08-27 10:30:31 -07:00
Arpit Agarwal
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 2019-08-26 15:57:55 -07:00
Chen Liang
ebef99dcf4 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:18:43 -07:00
Erik Krogen
d699022fce HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 2019-08-23 11:33:21 -07:00
Akira Ajisaka
abc8fde4ca
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 2019-08-23 14:44:34 +09:00
Stephen O'Donnell
b67812ea21 HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index (#1028). Contributed by Stephen O'Donnell. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-08-22 17:09:57 -07:00
Inigo Goiri
e04dcfdc57 HDFS-14583. FileStatus#toString() will throw IllegalArgumentException. Contributed by xuzq. 2019-08-22 10:22:38 -07:00
Ayush Saxena
0f598aed13 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 02:25:06 +05:30
Surendra Singh Lilhore
3a145e2918 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-20 15:55:26 -07:00
Wei-Chiu Chuang
360a96f342 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 2019-08-19 13:08:55 -07:00
HUAN-PING SU
abae6ff2a2 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-19 11:14:51 -07:00
Surendra Singh Lilhore
b8db5b9a98 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 18:13:35 -07:00
LeonGao91
9a1d8cfaf5 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 2019-08-16 08:00:51 -07:00
Shashikant Banerjee
0a85af959c HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 2019-08-15 10:16:25 +05:30
Siyao Meng
3c0382f1b9 HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2019-08-14 07:30:35 -07:00
Wei-Chiu Chuang
633b7c1cfe HDFS-14625. Make DefaultAuditLogger class in FSnamesystem to Abstract. Contributed by hemanthboyina. 2019-08-13 16:50:49 -07:00
Masatake Iwasaki
da0006fe04 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-14 08:39:40 +09:00
Ayush Saxena
e9b6b81de4 HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng. 2019-08-13 19:17:10 +05:30
Ayush Saxena
454420e4f2 HDFS-14708. TestLargeBlockReport#testBlockReportSucceedsWithLargerLengthLimit fails in trunk. Contributed by Lisheng Sun. 2019-08-13 08:27:20 +05:30
Siyao Meng
c92b49876a HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 2019-08-12 17:26:20 -07:00
Wei-Chiu Chuang
865021b8c9 HDFS-14195. OIV: print out storage policy id in oiv Delimited output. Contributed by Wang, Xinglong. 2019-08-09 15:37:29 -07:00
Wei-Chiu Chuang
2265872c2d HDFS-14705. Remove unused configuration dfs.min.replication. Contributed by CR Hota. 2019-08-08 13:48:29 -07:00
Stephen O'Donnell
b0799148cf HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-08 13:36:39 -07:00
Erik Krogen
827dbb11e2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 2019-08-07 09:25:58 -07:00
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Stephen O'Donnell
35e0a01d7b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 07:57:46 -07:00
Ayush Saxena
8f40856f76 HDFS-14669. TestDirectoryScanner#testDirectoryScannerInFederatedCluster fails intermittently in trunk. Contributed by qiang Liu. 2019-08-03 12:15:41 +05:30
Erik Krogen
e7a0b8aa83 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 2019-08-02 11:48:31 -07:00
Wei-Chiu Chuang
b964b81f85 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 2019-08-02 10:54:32 -07:00
Wei-Chiu Chuang
c2d00c8450 HDFS-13131. Modifying testcase testEnableAndDisableErasureCodingPolicy. Contributed by chencan. 2019-08-01 19:11:43 -07:00
Siyao Meng
99bf1dc9eb HDFS-14683. WebHDFS: Add erasureCodingPolicy field to GETCONTENTSUMMARY response (#1189) Contributed by Siyao Meng. 2019-08-01 17:14:07 -07:00
Wei-Chiu Chuang
32607dbd98 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-01 15:48:36 -07:00
Chao Sun
3ae775d740 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 16:01:17 -07:00
Erik Krogen
1f26cc8705 HDFS-13783. Add an option to the Balancer to make it run as a long-running service. Contributed by Chen Zhang. 2019-07-30 15:42:55 -07:00
Inigo Goiri
7849bdcf70 HDFS-14449. Expose total number of DT in JMX for Namenode. Contributed by Fengnan Li. 2019-07-30 13:45:27 -07:00
Ayush Saxena
8deced0073 HDFS-14677. TestDataNodeHotSwapVolumes#testAddVolumesConcurrently fails intermittently in trunk. Contributed by Chen Zhang. 2019-07-30 22:17:39 +05:30
Wei-Chiu Chuang
8053085388 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:31:34 -07:00
Ayush Saxena
02bd02b5af HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-07-28 08:41:42 +05:30
Ayush Saxena
ce99cc31e9 HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-07-26 08:31:00 +05:30
Masatake Iwasaki
6b8107ad97 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-07-26 09:35:23 +09:00
Ayush Saxena
62deab17a3 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 2019-07-25 06:51:07 +05:30
Chen Liang
5e6cc6fe8a HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-07-17 14:37:16 -07:00
Ayush Saxena
af0665c7f5 HDFS-13647. Fix the description of storageType option for space quota. Contributed by Takanobu Asanuma. 2019-07-17 23:22:32 +05:30
Ayush Saxena
f77d54c243 HDFS-14642. processMisReplicatedBlocks does not return correct processed count. Contributed by Stephen O'Donnell. 2019-07-16 08:14:27 +05:30
Rakesh Radhakrishnan
e98adb00b7 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 2019-07-15 13:02:37 +05:30
Shashikant Banerjee
f9fab9f22a HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 2019-07-12 15:42:06 +05:30
Masatake Iwasaki
00dd843a1a
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-12 10:04:55 +09:00
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Inigo Goiri
eccc9a40de HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-10 11:11:52 -07:00
Ajay Yadav
cdb20adfcc HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. (#973) 2019-07-10 11:03:58 -07:00
Yiqun Lin
993dc8726b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 2019-07-10 10:53:34 +08:00
Erik Krogen
4632708148 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-08 14:46:25 -07:00
Wei-Chiu Chuang
ec851e4db2 HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 2019-07-08 15:23:46 +08:00
Inigo Goiri
4a212242d9 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 2019-06-27 10:00:05 -07:00
Ajay Kumar
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Lokesh Jain
272b96d243 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:23:05 -07:00
Surendra Singh Lilhore
5962a518bd HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 09:40:29 -07:00
Wei-Chiu Chuang
71ecd2e411 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-19 10:27:53 -07:00
Wei-Chiu Chuang
d3ac516665 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit a1c3868c4f.
 2019-06-19 06:27:02 -07:00
Virajith Jalaparti
a1c3868c4f HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 22:17:49 -07:00
Elek, Marton
81ec909411 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 12:01:26 -07:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Wei-Chiu Chuang
335c1c9938 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 2019-06-18 00:35:48 -07:00
Oleg Danilov
54cdde38c7 HDFS-12314. Typo in the TestDataNodeHotSwapVolumes.testAddOneNewVolume(). Contributed by Oleg Danilov. 2019-06-17 23:11:25 -07:00
Oleg Danilov
d41310a15d HDFS-12315. Use Path instead of String to check closedFiles set. Contributed by Oleg Danilov. 2019-06-17 23:06:02 -07:00
Santosh Marella
6822193ee6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:20:30 -07:00
Wei-Chiu Chuang
a50c35bb81 Revert "HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit cdc5de6448.
 2019-06-17 16:18:37 -07:00
Wei-Chiu Chuang
f9a7b442fd HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-17 15:18:17 -07:00
Adam Antal
8370a0ae16 HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-15 06:47:10 -07:00
Santosh Marella
cdc5de6448 HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 13:01:39 -07:00
Inigo Goiri
4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Anu Engineer
101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
Inigo Goiri
e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri
944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
He Xiaoqiao
33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena
c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Eric Yang
d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Erik Krogen
55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Konstantin V Shvachko
c31b7b8d52 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:34:29 -07:00
Inigo Goiri
c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Inigo Goiri
1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Virajith Jalaparti
77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Inigo Goiri
389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Erik Krogen
5847e00143 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 2019-05-13 08:39:59 -07:00
Rakesh Radhakrishnan
9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore
69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Inigo Goiri
865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Sahil Takiar
4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00