big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,268 Commits 2 Branches 2 Tags 264 MiB
0b43c65816
Commit Graph

5895 Commits

Author SHA1 Message Date
S O'Donnell
97d6058aa6 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 15:37:52 +01:00
hemanthboyina
60d167d080 HDFS-14504. Rename with Snapshots does not honor quota limit. 
(cherry picked from commit 83fd15b412)
 2020-08-23 16:12:36 +05:30
Chen Liang
b1292bdaef HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d)
 2020-08-20 15:27:52 -07:00
S O'Donnell
42154f04e3 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 
(cherry picked from commit 10716040a8)
 2020-08-12 09:33:14 +01:00
hemanthboyina
b5d712251c HDFS-15503. File and directory permissions are not able to be modified from WebUI. 
(cherry picked from commit 82f3ffcd64)
 2020-08-03 23:43:29 +05:30
Uma Maheswara Rao G
aea1a8e2bd HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:30:30 -07:00
Chen Liang
c8c40be761 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 15:22:34 -07:00
Ahmed Hussein
9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Ahmed Hussein
5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 09:57:36 -05:00
Shashikant Banerjee
292e22578a HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 
(cherry picked from commit 82343790ee)
 2020-07-15 13:13:27 +01:00
He Xiaoqiao
e7e7a6d503 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:31:50 +08:00
Shanyu Zhao
10c9df1d0a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 11:44:42 +08:00
Ayush Saxena
b8d9218189 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:25:29 +05:30
Akira Ajisaka
3e9422d1c7
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:04:47 +09:00
Stephen O'Donnell
7613191fcd HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 630ac9e7e5fdbc3ce86476cf0167255ab9b0470a)
 2020-06-18 06:45:28 -07:00
Wei-Chiu Chuang
10880dc396 Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." 
This reverts commit 0b9e5ea592.
 2020-06-18 06:45:28 -07:00
S O'Donnell
3dc9db3aed HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 
(cherry picked from commit 123777823e)
 2020-06-18 12:29:12 +01:00
Ayush Saxena
bee2846bee HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7)
 2020-06-16 16:54:40 -07:00
Uma Maheswara Rao G
0b5e202614 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
 2020-06-16 16:53:38 -07:00
Stephen O'Donnell
0b9e5ea592 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 730a39d138)
 2020-06-16 16:02:07 -07:00
Takanobu Asanuma
3b8418250f HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077)
 2020-06-15 09:17:40 +09:00
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Toshihiro Suzuki
ec8f3714e0 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)
 2020-06-05 11:17:13 +01:00
Wanqiang Ji
9b84a637b7
HADOOP-17055. Remove residual code of Ozone (#2039) 
(cherry picked from commit d9838f2d42)
 2020-05-29 16:50:10 +09:00
Ayush Saxena
6b7040a1cb HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:04:22 +05:30
Ayush Saxena
718c8a5868 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:06:19 +05:30
Takanobu Asanuma
c44d8b8bdf HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:33:39 +09:00
Chen Liang
e452163a06
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 
(cherry picked from commit 7bb902bc0d)
 2020-05-19 09:52:25 +09:00
He Xiaoqiao
eb045ea056 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 09:59:12 -07:00
Ayush Saxena
3915d1afc7 HDFS-15356. Unify configuration dfs.ha.allow.stale.reads to DFSConfigKeys. Contributed by Xiaoqiao He. 
(cherry picked from commit 178336f8a8)
 2020-05-18 09:58:59 -07:00
Wei-Chiu Chuang
240cba7e6d HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
(cherry picked from commit 2abcf7762a)
 2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang
289f8acc64 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit ad9a6a0ee3.
 2020-05-18 09:41:03 -07:00
Wei-Chiu Chuang
53d22fdb88 Revert "HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He." 
This reverts commit acae31aa28.
 2020-05-18 09:39:57 -07:00
Wei-Chiu Chuang
032ccba67c HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 
(cherry picked from commit 50caba1a92)
 2020-05-18 08:42:20 -07:00
He Xiaoqiao
acae31aa28 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 07:10:27 -07:00
pustota2009
ad9a6a0ee3 HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

(cherry picked from commit 86e6aa8eec)
 2020-05-18 07:05:03 -07:00
S O'Donnell
433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena
ef41229f70 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:43:37 +05:30
Ayush Saxena
61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:45:18 -07:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9a)
 2020-04-29 16:01:48 +01:00
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:18:25 +09:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:06:08 -07:00
Wei-Chiu Chuang
d6c48f8100 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
 2020-04-24 11:54:11 -07:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Stephen O'Donnell
aaad947c74 HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a)
 2020-04-16 20:19:00 -07:00
Wei-Chiu Chuang
749a5b81da HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 061afcdf30)
 2020-04-09 09:45:05 -07:00
Inigo Goiri
f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena
696a663cd0 HDFS-15193. Improving the error message for missing dfs.namenode.rpc-address.$NAMESERVICE. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Wei-Chiu Chuang
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909) 2020-03-25 16:03:26 -07:00
Inigo Goiri
cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Arpit Agarwal
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. 
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
 2020-03-25 10:28:30 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Ayush Saxena
ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Karthik Palanisamy
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
Ayush Saxena
62c8015683 HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena. 2020-03-22 11:44:18 +05:30
Takanobu Asanuma
b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
Ayush Saxena
f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena
bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Ayush Saxena
654db35fa2 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-03-18 21:00:45 +05:30
Ayush Saxena
8a7c54995a HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina. 2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang
7b637a3341 HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun. 2020-03-13 19:01:23 -07:00
He Xiaoqiao
e9955bb8ff HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829) 
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
 2020-03-13 11:29:44 -07:00
Ayush Saxena
fb1d7635ae HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang. 2020-03-13 21:10:03 +05:30
S O'Donnell
25497799cb HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang
20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Inigo Goiri
3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Ayush Saxena
1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Surendra Singh Lilhore
fccfb02672 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:33:06 +05:30
Ayush Saxena
429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Surendra Singh Lilhore
2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena
033a3d7ff9 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:26 +05:30
Ayush Saxena
209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang
1c5d2f1fdc HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 2020-02-24 13:39:16 -08:00
Ayush Saxena
7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena
93b8f453b9 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:01 +05:30
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena
9eb7a8bdf8 HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi. 2020-02-23 14:25:39 +05:30
Konstantin V Shvachko
ecbcb058b8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 2020-02-21 17:53:37 -08:00
Konstantin V Shvachko
6f84269bcd HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 2020-02-21 10:51:14 -08:00
Masatake Iwasaki
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 2020-02-21 11:56:07 +09:00
Inigo Goiri
ec7507162c HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham. 2020-02-19 11:33:58 -08:00
Ayush Saxena
ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Surendra Singh Lilhore
810783d443 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-02-16 12:15:03 +05:30
Surendra Singh Lilhore
a98352ced1 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 16:57:41 +05:30
Akira Ajisaka
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router (#1832) 
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
 2020-02-13 10:06:07 +09:00
Kihwal Lee
9b8a78d97b HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.
 2020-02-11 12:40:00 -06:00
Stephen O'Donnell
d7c136b9ed HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-11 08:00:15 -08:00
Ayush Saxena
6191d4b4a0 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 2020-02-09 23:32:22 +05:30
Ayush Saxena
d23317b102 HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:33:57 +05:30