big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,450 Commits 2 Branches 2 Tags 264 MiB
3363cc380b
Commit Graph

5957 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze
41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. (#2233) 2020-08-27 14:54:52 +05:30
Stephen O'Donnell
2ffe00fc46 HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-08-26 23:05:32 -07:00
S O'Donnell
6a49bf9bff HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 2020-08-25 15:18:36 +01:00
He Xiaoqiao
a7830423c5 HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu. 2020-08-24 21:16:00 +08:00
hemanthboyina
83fd15b412 HDFS-14504. Rename with Snapshots does not honor quota limit. 2020-08-23 16:10:47 +05:30
Chen Liang
f734455e5d HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 15:25:10 -07:00
Uma Maheswara Rao G
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 2020-08-19 09:30:41 -07:00
Siyao Meng
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS (#2220) 2020-08-18 19:28:19 +09:00
bshashikant
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. (#2172) 
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.

* Addressed review comments.

* Rebased to latest trunk and added snapshotTrashRoot config check.

* Addressed review comments.

* Removede empty line added in SnapshotManager.Java.

* Addressed whitespace issues.
 2020-08-17 12:56:13 +05:30
Ayush Saxena
e3d1966f58 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 2020-08-15 15:07:58 +05:30
Tsz-Wo Nicholas Sze
b93dd7c281
HDFS-15519. Check inaccessible INodes in FsImageValidation. (#2224) 2020-08-14 10:10:01 -07:00
bshashikant
15a76e8d65
HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. (#2219) 2020-08-14 20:46:48 +05:30
Vivek Ratnavel Subramanian
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots (#2212) 2020-08-13 22:36:15 +05:30
Aryan Gupta
e592ec5f8b
HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. (#2217) 
* HDFS-15518. Fixed String operationName = ListSnapshot.

* HDFS-15518. Changed ListSnapshot to operationName.

Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>
 2020-08-13 00:09:38 +05:30
S O'Donnell
10716040a8 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 2020-08-12 09:02:47 +01:00
Tsz-Wo Nicholas Sze
11cec9ab94
HDFS-15523. Fix findbugs warnings from HDFS-15520. (#2218) 2020-08-11 21:41:28 -07:00
Siyao Meng
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory (#2176) 2020-08-11 08:52:16 -07:00
Tsz-Wo Nicholas Sze
592127bdc2
HDFS-15520 Use visitor pattern to visit namespace tree (#2203) 2020-08-11 12:30:30 +05:30
Akira Ajisaka
7938ebfb9d
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 2020-08-11 11:14:58 +09:00
Ayush Saxena
c2a17659d1 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 2020-08-08 14:33:53 +05:30
bshashikant
e072d33327
HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable (#2175) 2020-08-04 14:10:29 +05:30
hemanthboyina
82f3ffcd64 HDFS-15503. File and directory permissions are not able to be modified from WebUI. 2020-08-03 23:21:01 +05:30
bshashikant
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181) 2020-08-02 21:59:56 +05:30
hemanthboyina
528a799a78 HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree. 2020-08-02 12:02:37 +05:30
Tsz-Wo Nicholas Sze
05b3337a46
HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots (#2165) 2020-07-30 23:06:51 +05:30
bshashikant
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166) 2020-07-29 21:33:25 +05:30
touchida
026dce5334
HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. (#2135) 2020-07-27 09:55:11 -07:00
bshashikant
2d12496643
HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (#2163) 2020-07-22 16:16:27 -07:00
Masatake Iwasaki
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Inigo Goiri
8b7695bb26 HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 2020-07-21 09:00:07 -07:00
Tsz-Wo Nicholas Sze
d57462f2da
HDFS-15479. Ordered snapshot deletion: make it a configurable feature (#2156) 2020-07-21 11:36:24 +05:30
Chen Liang
3833c616e0 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 2020-07-20 12:49:58 -07:00
Ayush Saxena
6cbd8854ee HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi. 2020-07-20 21:49:05 +05:30
Tsz-Wo Nicholas Sze
2cec50cf16
HDFS-15463. Add a tool to validate FsImage. (#2140) 2020-07-20 11:44:30 +05:30
Ahmed Hussein
1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 11:39:06 -05:00
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Shashikant Banerjee
85d4718ed7 HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee. 2020-07-15 10:32:25 +05:30
zhaoyim
bdce75d737
HDFS-15371. Nonstandard characters exist in NameNode.java (#2032) 
Contributed by zhaoyim
 2020-07-14 14:42:12 +01:00
He Xiaoqiao
b97fea65e7 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:12:48 +08:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Ye Ni
f77bbc2123
HDFS-15312. Apply umask when creating directory by WebHDFS (#2096) 2020-07-06 19:01:46 -07:00
Shanyu Zhao
834372f404
HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-06 23:43:34 +08:00
Uma Maheswara Rao G
55a2ae80dc
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 2020-07-04 13:45:49 -07:00
Ayush Saxena
f86f15cf20 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 13:00:43 +05:30
He Xiaoqiao
9ac498e300 HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu. 2020-07-01 12:38:07 +08:00
Stephen O'Donnell
2a67e2b1a0 HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-30 07:10:17 -07:00
Akira Ajisaka
c71ce7ac33
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 2020-06-28 16:02:47 +09:00
Stephen O'Donnell
d50e93ce7b HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang
edf716a5c3 Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." 
This reverts commit 730a39d138.
 2020-06-18 06:45:15 -07:00
S O'Donnell
123777823e HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 2020-06-18 12:26:22 +01:00
Stephen O'Donnell
730a39d138 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-16 15:59:13 -07:00
Takanobu Asanuma
f41a144077 HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 2020-06-15 09:15:53 +09:00
Inigo Goiri
719b53a79d HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 2020-06-13 09:35:05 -07:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Ayush Saxena
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 2020-06-05 22:19:38 -07:00
Ayush Saxena
2326123705 HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena. 2020-06-05 19:15:05 +05:30
Toshihiro Suzuki
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.
 2020-06-05 11:11:49 +01:00
Ayush Saxena
e8cb2ae409 HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 2020-06-03 12:47:15 +05:30
Uma Maheswara Rao G
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00
Ayush Saxena
ae13a5ccbe HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 2020-05-31 17:10:59 +05:30
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Ayush Saxena
9b38be43c6 HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:00:04 +05:30
Inigo Goiri
2148a8fe64 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 2020-05-27 11:06:13 -07:00
Ayush Saxena
6c9f75cf16 HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena. 2020-05-26 16:26:57 +05:30
Ayush Saxena
f43a152b97 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:03:41 +05:30
Ayush Saxena
f4901d0778 HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun. 2020-05-24 17:59:16 +05:30
Inigo Goiri
6e04b00df1 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 2020-05-23 09:58:19 -07:00
Ayush Saxena
e0ae232f66 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 2020-05-23 18:38:32 +05:30
Ayush Saxena
f99fcb26ab HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2020-05-23 18:29:31 +05:30
Takanobu Asanuma
4d22d1c58f HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 2020-05-23 17:28:38 +09:00
leosunli
be374faf42
HDFS-13639. SlotReleaser is not fast enough (#1885) 2020-05-21 13:21:17 -07:00
Chen Liang
7bb902bc0d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 10:58:52 -07:00
Ayush Saxena
c84e6beada HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 2020-05-18 22:24:34 +05:30
Wei-Chiu Chuang
2abcf7762a HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang
4525292d41 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit 86e6aa8eec.
 2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang
50caba1a92 HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 2020-05-18 08:40:38 -07:00
He Xiaoqiao
a3f44dacc1 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-18 07:09:43 -07:00
pustota2009
86e6aa8eec
HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
 2020-05-18 07:04:04 -07:00
Ayush Saxena
6e416a83d1 HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 2020-05-17 03:06:27 +05:30
Ayush Saxena
178336f8a8 HDFS-15356. Unify configuration dfs.ha.allow.stale.reads to DFSConfigKeys. Contributed by Xiaoqiao He. 2020-05-16 16:35:06 +05:30
Xiaoyu Yao
3cacf1ce56
HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. (#2004) 2020-05-13 11:47:19 -07:00
Surendra Singh Lilhore
743c2e9071 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 2020-05-13 15:01:07 +05:30
Elixir Kook
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007) 
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
 2020-05-12 10:50:04 -05:00
S O'Donnell
29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Takanobu Asanuma
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. (#2008) 2020-05-12 09:04:03 -05:00
Ayush Saxena
8dad38c0be HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:34:26 +05:30
Ayush Saxena
0fe49036e5 HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu. 2020-05-12 13:11:31 +05:30
Ayush Saxena
4c53fb9ce1 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:48:34 +05:30
Ayush Saxena
cb64e993c2 HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi. 2020-05-10 00:11:46 +05:30
Inigo Goiri
130f89e068 HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 2020-05-06 13:18:32 -07:00
Ayush Saxena
7fddf4855e HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 16:55:04 +05:30
Wei-Chiu Chuang
adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
 2020-05-05 16:04:54 -07:00
Konstantin V Shvachko
ebb878bab9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 2020-05-04 10:29:50 -07:00
Ayush Saxena
8dace8ff3a HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi. 2020-05-03 23:59:17 +05:30
Ayush Saxena
d6fc482a54 Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:09:14 +05:30
Akira Ajisaka
f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:02:27 -07:00
Shashikant Banerjee
82343790ee HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-05-01 12:16:02 +05:30
Mingliang Liu
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-04-30 14:15:28 -07:00
S O'Donnell
9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
Akira Ajisaka
5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 2020-04-28 06:17:14 +09:00