big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,812 Commits 2 Branches 2 Tags 264 MiB
043628dcf1
Commit Graph

23812 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
S O'Donnell
433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena
ef41229f70 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:43:37 +05:30
Szilard Nemeth
95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Babneet Singh
d1e5e393c3 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
 2020-05-11 10:45:57 -07:00
Ayush Saxena
61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Akira Ajisaka
282427f6d1
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003) 
(cherry picked from commit 328eae9a14)
 2020-05-11 14:46:43 +09:00
Ayush Saxena
6ccb1cd5f1 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Akira Ajisaka
763a79916d
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 
(cherry picked from commit c784ba370e)
 2020-05-08 17:19:48 +09:00
Ayush Saxena
817814105a HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:06 +05:30
Masatake Iwasaki
6a0aeea60f HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994) 
(cherry picked from commit 99840aaba6)
 2020-05-07 12:02:15 +09:00
Uma Maheswara Rao G
edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-06 15:13:33 -07:00
Ahmed Hussein
7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:28:04 -05:00
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:45:18 -07:00
Ahmed Hussein
1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:24:01 -05:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
Eric Yang
0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:51:12 +09:00
Ayush Saxena
ccebc9d9d0 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-05-01 14:43:34 +05:30
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9a)
 2020-04-29 16:01:48 +01:00
Szilard Nemeth
8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth
75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Aryan Gupta
08dbe32122 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 
(cherry picked from commit 816042e62b)
 2020-04-29 11:16:56 +05:30
Szilard Nemeth
e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:18:25 +09:00
Akira Ajisaka
62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba)
 2020-04-28 05:45:37 +09:00
Eric E Payne
741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117)
 2020-04-27 19:34:38 +00:00
Mike
68d8802624 HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952) 
Contributed by Mikhail Pryakhin.
 2020-04-27 14:46:52 +01:00
Szilard Nemeth
c0b7b38e22 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 
(cherry picked from commit f91e21ac10)
 2020-04-27 08:26:46 +02:00
Ayush Saxena
9e79cca171 HADOOP-17007. hadoop-cos fails to build. Contributed by Yang Yu. 2020-04-26 12:55:55 +05:30
Mingliang Liu
1db1992b31
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:44:01 -07:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:06:08 -07:00
Ayush Saxena
c2769384ac HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-04-25 13:23:16 +05:30
Wei-Chiu Chuang
d6c48f8100 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
 2020-04-24 11:54:11 -07:00
Szilard Nemeth
7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth
61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka
497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961)
 2020-04-24 10:27:39 +09:00
Mehakmeet Singh
2471ba8b5c HADOOP-16914 Adding Output Stream Counters in ABFS (#1899) 
Contributed by Mehakmeet Singh.There
 2020-04-23 14:38:15 +01:00
Sneha Vijayarajan
32fb174da2 Hadoop 16857. ABFS: Stop CustomTokenProvider retry logic to depend on AbfsRestOp retry policy 
Contributed by Sneha Vijayarajan
 2020-04-23 14:37:25 +01:00
Szilard Nemeth
1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Mingliang Liu
5b92d73a74
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:48:39 -07:00
bilaharith
f53ded6185 HADOOP-16922. ABFS: Change User-Agent header (#1938) 
Contributed by Bilahari T H.
 2020-04-22 14:32:30 +01:00
Mukund Thakur
98fdbb820e HADOOP-16965. Refactor abfs stream configuration. (#1956) 
Contributed by Mukund Thakur.
 2020-04-22 14:32:01 +01:00
Mehakmeet Singh
f74a571fdf HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-04-22 14:30:25 +01:00
Akira Ajisaka
54a64e542e
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917) 
(cherry picked from commit 5576915236)
 2020-04-21 21:12:02 +09:00
Akira Ajisaka
5459dd64fd
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 
(cherry picked from commit e069a06137)
 2020-04-21 10:27:27 +09:00
Steve Loughran
0982f56f3a
HADOOP-16953. tuning s3guard disabled warnings (#1962) 
Contributed by Steve Loughran.

The S3Guard absence warning of HADOOP-16484 has been changed
so that by default the S3A connector only logs at debug
when the connection to the S3 Store does not have S3Guard
enabled.

The option to control this log level is now
fs.s3a.s3guard.disabled.warn.level
and can be one of: silent, inform, warn, fail.

On a failure, an ExitException is raised with exit code 49.

For details on this safety feature, consult the s3guard documentation.

Change-Id: If868671c9260977c2b03b3e475b9c9531c98ce79
 2020-04-20 15:07:00 +01:00