big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,911 Commits 2 Branches 2 Tags 264 MiB
e7dd02768b
Commit Graph

8837 Commits

Author SHA1 Message Date
Ayush Saxena
7fddf4855e HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 16:55:04 +05:30
Wei-Chiu Chuang
adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
 2020-05-05 16:04:54 -07:00
Uma Maheswara Rao G
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 2020-05-04 17:55:40 -07:00
Babneet Singh
1996351b0b
HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
 2020-05-04 13:08:53 -07:00
Konstantin V Shvachko
ebb878bab9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 2020-05-04 10:29:50 -07:00
Ayush Saxena
8dace8ff3a HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi. 2020-05-03 23:59:17 +05:30
Ayush Saxena
d6fc482a54 Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:09:14 +05:30
Mingliang Liu
44de193bec
HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li 2020-05-02 20:56:29 -07:00
Akira Ajisaka
f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:02:27 -07:00
Eric Yang
ba66f3b454
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-05-02 21:50:13 +09:00
Shashikant Banerjee
82343790ee HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-05-01 12:16:02 +05:30
Mingliang Liu
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-04-30 14:15:28 -07:00
Akira Ajisaka
b5b45c53a4
HDFS-15309. Remove redundant String.valueOf method on ExtendedBlockId.java. Contributed by bianqi. 2020-04-30 15:46:46 +09:00
Inigo Goiri
31b2f687ef HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina. 2020-04-29 13:41:36 -07:00
S O'Donnell
9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
Surendra Singh Lilhore
db6252b6c3 HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 2020-04-29 10:58:35 +05:30
Aryan Gupta
816042e62b
HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 2020-04-29 09:19:59 +05:30
Akira Ajisaka
5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 2020-04-28 06:17:14 +09:00
Toshihiro Suzuki
62c26b91fd
HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 (#1979) 2020-04-27 17:20:47 +09:00
Mingliang Liu
74d4a18735
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:43:02 -07:00
Dhiraj
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:04:32 -07:00
Wei-Chiu Chuang
453771f170 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 2020-04-24 11:53:24 -07:00
Mingliang Liu
5958af4dbc
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:31:02 -07:00
Akira Ajisaka
e069a06137
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 2020-04-20 08:41:18 -07:00
Takanobu Asanuma
1fdfaebd98 HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina. 2020-04-20 23:09:33 +09:00
Toshihiro Suzuki
1824aee9da
HDFS-15217 Add more information to longest write/read lock held log 2020-04-18 13:52:07 -07:00
Surendra Singh Lilhore
8e6227441a HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 2020-04-18 20:07:21 +05:30
Ayush Saxena
37d6582223 HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 2020-04-17 13:32:02 +05:30
Stephen O'Donnell
3481895f8a HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-04-16 20:18:25 -07:00
Ayush Saxena
cc5c1da7c1 HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang. 2020-04-15 23:01:58 +05:30
Ayush Saxena
8d49229c37 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-04-12 12:05:50 +05:30
Ayush Saxena
275c478330 HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena. 2020-04-09 23:44:36 +05:30
Wei-Chiu Chuang
061afcdf30
HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-04-09 09:19:35 -07:00
Ayush Saxena
1189af4746 HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena. 2020-04-08 15:55:14 +05:30
Toshihiro Suzuki
c12ddbd1de
HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 2020-04-07 14:38:09 +09:00
Inigo Goiri
0b855b9f35 HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun. 2020-04-06 13:05:33 -07:00
Akira Ajisaka
1695d8d59c
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 2020-04-04 00:13:41 +09:00
Takanobu Asanuma
736659e0e1 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 2020-04-02 19:21:02 +09:00
Ayush Saxena
4a3eb10972 HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He. 2020-04-02 11:06:01 +05:30
He Xiaoqiao
d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-01 16:36:46 -07:00
Inigo Goiri
80b877a72f HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 2020-03-30 12:29:21 -07:00
Ayush Saxena
960c9ebaea HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 2020-03-30 02:40:25 +05:30
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Inigo Goiri
f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena
696a663cd0 HDFS-15193. Improving the error message for missing dfs.namenode.rpc-address.$NAMESERVICE. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Chen Liang
f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00
Ayush Saxena
cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Inigo Goiri
679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
Wei-Chiu Chuang
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909) 2020-03-25 16:03:26 -07:00
Inigo Goiri
cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Arpit Agarwal
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. 
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
 2020-03-25 10:28:30 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Isa Hekmatizadeh
d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 2020-03-24 23:17:22 +05:30
Ayush Saxena
ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Karthik Palanisamy
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
He Xiaoqiao
5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 22:41:31 -07:00
Akira Ajisaka
f59f6891c8
HDFS-15232. Fix libhdfspp test failures with GCC 7. (#1906) 2020-03-24 11:36:14 +09:00
He Xiaoqiao
af64ce2f4a HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 12:43:51 -07:00
Ayush Saxena
62c8015683 HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena. 2020-03-22 11:44:18 +05:30
Takanobu Asanuma
b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
Ayush Saxena
f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena
bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Ayush Saxena
654db35fa2 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
 2020-03-18 13:27:13 +00:00
Surendra Singh Lilhore
1d772dc542 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:44:32 +05:30
Ayush Saxena
74780c22eb HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:30:39 +05:30
Ayush Saxena
8a7c54995a HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina. 2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang
7b637a3341 HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun. 2020-03-13 19:01:23 -07:00
He Xiaoqiao
e9955bb8ff HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829) 
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
 2020-03-13 11:29:44 -07:00
Ayush Saxena
fb1d7635ae HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang. 2020-03-13 21:10:03 +05:30
S O'Donnell
25497799cb HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang
20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface 
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
 2020-03-09 14:51:16 +00:00
Steve Loughran
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden 
Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
 2020-03-09 14:44:28 +00:00
Inigo Goiri
4062217189 HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena. 2020-03-06 10:14:53 -08:00
Inigo Goiri
3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Inigo Goiri
1d4d0fcbe1 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 2020-03-04 10:02:54 -08:00
Akshay Nehe
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871) 2020-03-04 11:07:31 +09:00
Xiaoyu Yao
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file 
Contributed by Xiaoyu Yao.

Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails

Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.

This is a convergence of a fix for S3 meeting an issue in HDFS.

S3: a HEAD against a file can cache a 404, 
 -you must not do any existence checks, including deleteOnExit(),
  until the file is written. 

Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct. 

HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.

This patch:

* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
 2020-03-02 13:22:00 +00:00
Ayush Saxena
1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Inigo Goiri
97b797c314 HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun. 2020-02-28 18:47:22 -08:00
Takanobu Asanuma
9a3a28018a HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina. 2020-02-29 10:30:26 +09:00
Surendra Singh Lilhore
fccfb02672 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:33:06 +05:30
Ayush Saxena
429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Istvan Fajth
57aa048516
HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 2020-02-27 07:48:14 -08:00
Surendra Singh Lilhore
2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena
033a3d7ff9 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:26 +05:30
Ayush Saxena
209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang
1c5d2f1fdc HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 2020-02-24 13:39:16 -08:00
Ayush Saxena
7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena
93b8f453b9 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:01 +05:30
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena
9eb7a8bdf8 HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi. 2020-02-23 14:25:39 +05:30
Ayush Saxena
ba9025c7cd HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun. 2020-02-22 14:27:26 +05:30