23810 Commits

Author SHA1 Message Date
Szilard Nemeth
95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Babneet Singh
d1e5e393c3 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951)
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b7be6866eda73223ab6ef1ec78d30cd)
2020-05-11 10:45:57 -07:00
Ayush Saxena
61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Akira Ajisaka
282427f6d1
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)
(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)
2020-05-11 14:46:43 +09:00
Ayush Saxena
6ccb1cd5f1 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Akira Ajisaka
763a79916d
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001)
(cherry picked from commit c784ba370eeb3e48c1c2c471ca72674c62a4edb7)
2020-05-08 17:19:48 +09:00
Ayush Saxena
817814105a HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:06 +05:30
Masatake Iwasaki
6a0aeea60f HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994)
(cherry picked from commit 99840aaba662e7bb6187206b74e35132102a3b38)
2020-05-07 12:02:15 +09:00
Uma Maheswara Rao G
edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
2020-05-06 15:13:33 -07:00
Ahmed Hussein
7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
2020-05-06 12:28:04 -05:00
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998)
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b534c536354f4f47789467ffb82dd1496)
2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.
(cherry picked from commit ebb878bab991c242b5089a18881aa10abf318ea0)
2020-05-04 10:45:18 -07:00
Ahmed Hussein
1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
2020-05-04 10:24:01 -05:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-02 13:59:58 -07:00
Eric Yang
0f27c04c23
SPNEGO TLS verification
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
2020-05-02 21:51:12 +09:00
Ayush Saxena
ccebc9d9d0 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-05-01 14:43:34 +05:30
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.
(cherry picked from commit 9ca6298a9ac1ffd5c87222d5b81a561a86913840)
2020-04-29 16:01:48 +01:00
Szilard Nemeth
8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G
(cherry picked from commit 410c605aec308a2ccd903f60aade3aaeefcaa610)
2020-04-29 15:54:32 +02:00
Szilard Nemeth
75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Aryan Gupta
08dbe32122 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980)
(cherry picked from commit 816042e62bf472a58d9f6dbce1123e9af6d06fb0)
2020-04-29 11:16:56 +05:30
Szilard Nemeth
e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori
(cherry picked from commit ab3642955971dec1ce285f39cf0f02e6cc64b4b2)
2020-04-28 18:56:29 +02:00
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.
(cherry picked from commit 5e0eda5d5f696aba7fc209874d232baf2a50d547)
2020-04-28 06:18:25 +09:00
Akira Ajisaka
62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand.
(cherry picked from commit 6ef01646ba9540248ffaf322f4678fbdac28fcd4)
2020-04-28 05:45:37 +09:00
Eric E Payne
741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
(cherry picked from commit e2322e1117a2a3435aabecd49de0ad3c8d2f64b9)
2020-04-27 19:34:38 +00:00
Mike
68d8802624 HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952)
Contributed by Mikhail Pryakhin.
2020-04-27 14:46:52 +01:00
Szilard Nemeth
c0b7b38e22 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph
(cherry picked from commit f91e21ac109e753e76d19c5c872c59a767b7b837)
2020-04-27 08:26:46 +02:00
Ayush Saxena
9e79cca171 HADOOP-17007. hadoop-cos fails to build. Contributed by Yang Yu. 2020-04-26 12:55:55 +05:30
Mingliang Liu
1db1992b31
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:44:01 -07:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964)
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:06:08 -07:00
Ayush Saxena
c2769384ac HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-04-25 13:23:16 +05:30
Wei-Chiu Chuang
d6c48f8100 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.
(cherry picked from commit 453771f170a062265b7687b31b2a3735827958fb)
2020-04-24 11:54:11 -07:00
Szilard Nemeth
7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth
61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka
497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939)
(cherry picked from commit 9827ff2961e249e8be9023f39d91e1c0fbd23d61)
2020-04-24 10:27:39 +09:00
Mehakmeet Singh
2471ba8b5c HADOOP-16914 Adding Output Stream Counters in ABFS (#1899)
Contributed by Mehakmeet Singh.There
2020-04-23 14:38:15 +01:00
Sneha Vijayarajan
32fb174da2 Hadoop 16857. ABFS: Stop CustomTokenProvider retry logic to depend on AbfsRestOp retry policy
Contributed by Sneha Vijayarajan
2020-04-23 14:37:25 +01:00
Szilard Nemeth
1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Mingliang Liu
5b92d73a74
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:48:39 -07:00
bilaharith
f53ded6185 HADOOP-16922. ABFS: Change User-Agent header (#1938)
Contributed by Bilahari T H.
2020-04-22 14:32:30 +01:00
Mukund Thakur
98fdbb820e HADOOP-16965. Refactor abfs stream configuration. (#1956)
Contributed by Mukund Thakur.
2020-04-22 14:32:01 +01:00
Mehakmeet Singh
f74a571fdf HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-04-22 14:30:25 +01:00
Akira Ajisaka
54a64e542e
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917)
(cherry picked from commit 5576915236aba172cb5ab49b43111661590058af)
2020-04-21 21:12:02 +09:00
Akira Ajisaka
5459dd64fd
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky
(cherry picked from commit e069a0613710926905de991d1d563cea5b4ee6df)
2020-04-21 10:27:27 +09:00
Steve Loughran
0982f56f3a
HADOOP-16953. tuning s3guard disabled warnings (#1962)
Contributed by Steve Loughran.

The S3Guard absence warning of HADOOP-16484 has been changed
so that by default the S3A connector only logs at debug
when the connection to the S3 Store does not have S3Guard
enabled.

The option to control this log level is now
fs.s3a.s3guard.disabled.warn.level
and can be one of: silent, inform, warn, fail.

On a failure, an ExitException is raised with exit code 49.

For details on this safety feature, consult the s3guard documentation.

Change-Id: If868671c9260977c2b03b3e475b9c9531c98ce79
2020-04-20 15:07:00 +01:00
Steve Loughran
de9a6b4588
HADOOP-16986. S3A to not need wildfly on the classpath. (#1948)
Contributed by Steve Loughran.

This is a successor to HADOOP-16346, which enabled the S3A connector
to load the native openssl SSL libraries for better HTTPS performance.

That patch required wildfly.jar to be on the classpath. This
update:

* Makes wildfly.jar optional except in the special case that
"fs.s3a.ssl.channel.mode" is set to "openssl"

* Retains the declaration of wildfly.jar as a compile-time
dependency in the hadoop-aws POM. This means that unless
explicitly excluded, applications importing that published
maven artifact will, transitively, add the specified
wildfly JAR into their classpath for compilation/testing/
distribution.

This is done for packaging and to offer that optional
speedup. It is not mandatory: applications importing
the hadoop-aws POM can exclude it if they choose.

Change-Id: I7ed3e5948d1e10ce21276b3508871709347e113d
2020-04-20 14:42:36 +01:00
Sammi Chen
9c81b17153 HADOOP-16959. Resolve hadoop-cos dependency conflict. Contributed by Yang Yu.
(cherry picked from commit 82ff7bc9abc8f3ad549db898953d98ef142ab02d)
2020-04-20 18:57:49 +08:00