big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,053 Commits 2 Branches 2 Tags 264 MiB
0712505b59
Commit Graph

24053 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Masatake Iwasaki
6a0aeea60f HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994) 
(cherry picked from commit 99840aaba6)
 2020-05-07 12:02:15 +09:00
Uma Maheswara Rao G
edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-06 15:13:33 -07:00
Ahmed Hussein
7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:28:04 -05:00
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:45:18 -07:00
Ahmed Hussein
1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:24:01 -05:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
Eric Yang
0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:51:12 +09:00
Ayush Saxena
ccebc9d9d0 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-05-01 14:43:34 +05:30
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9a)
 2020-04-29 16:01:48 +01:00
Szilard Nemeth
8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth
75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Aryan Gupta
08dbe32122 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 
(cherry picked from commit 816042e62b)
 2020-04-29 11:16:56 +05:30
Szilard Nemeth
e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:18:25 +09:00
Akira Ajisaka
62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba)
 2020-04-28 05:45:37 +09:00
Eric E Payne
741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117)
 2020-04-27 19:34:38 +00:00
Mike
68d8802624 HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952) 
Contributed by Mikhail Pryakhin.
 2020-04-27 14:46:52 +01:00
Szilard Nemeth
c0b7b38e22 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 
(cherry picked from commit f91e21ac10)
 2020-04-27 08:26:46 +02:00
Ayush Saxena
9e79cca171 HADOOP-17007. hadoop-cos fails to build. Contributed by Yang Yu. 2020-04-26 12:55:55 +05:30
Mingliang Liu
1db1992b31
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:44:01 -07:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:06:08 -07:00
Ayush Saxena
c2769384ac HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-04-25 13:23:16 +05:30
Wei-Chiu Chuang
d6c48f8100 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
 2020-04-24 11:54:11 -07:00
Szilard Nemeth
7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth
61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka
497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961)
 2020-04-24 10:27:39 +09:00
Mehakmeet Singh
2471ba8b5c HADOOP-16914 Adding Output Stream Counters in ABFS (#1899) 
Contributed by Mehakmeet Singh.There
 2020-04-23 14:38:15 +01:00
Sneha Vijayarajan
32fb174da2 Hadoop 16857. ABFS: Stop CustomTokenProvider retry logic to depend on AbfsRestOp retry policy 
Contributed by Sneha Vijayarajan
 2020-04-23 14:37:25 +01:00
Szilard Nemeth
1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Mingliang Liu
5b92d73a74
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:48:39 -07:00
bilaharith
f53ded6185 HADOOP-16922. ABFS: Change User-Agent header (#1938) 
Contributed by Bilahari T H.
 2020-04-22 14:32:30 +01:00
Mukund Thakur
98fdbb820e HADOOP-16965. Refactor abfs stream configuration. (#1956) 
Contributed by Mukund Thakur.
 2020-04-22 14:32:01 +01:00
Mehakmeet Singh
f74a571fdf HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-04-22 14:30:25 +01:00
Akira Ajisaka
54a64e542e
HADOOP-16944. Use Yetus 0.12.0 in GitHub PR (#1917) 
(cherry picked from commit 5576915236)
 2020-04-21 21:12:02 +09:00
Akira Ajisaka
5459dd64fd
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 
(cherry picked from commit e069a06137)
 2020-04-21 10:27:27 +09:00
Steve Loughran
0982f56f3a
HADOOP-16953. tuning s3guard disabled warnings (#1962) 
Contributed by Steve Loughran.

The S3Guard absence warning of HADOOP-16484 has been changed
so that by default the S3A connector only logs at debug
when the connection to the S3 Store does not have S3Guard
enabled.

The option to control this log level is now
fs.s3a.s3guard.disabled.warn.level
and can be one of: silent, inform, warn, fail.

On a failure, an ExitException is raised with exit code 49.

For details on this safety feature, consult the s3guard documentation.

Change-Id: If868671c9260977c2b03b3e475b9c9531c98ce79
 2020-04-20 15:07:00 +01:00
Steve Loughran
de9a6b4588
HADOOP-16986. S3A to not need wildfly on the classpath. (#1948) 
Contributed by Steve Loughran.

This is a successor to HADOOP-16346, which enabled the S3A connector
to load the native openssl SSL libraries for better HTTPS performance.

That patch required wildfly.jar to be on the classpath. This
update:

* Makes wildfly.jar optional except in the special case that
"fs.s3a.ssl.channel.mode" is set to "openssl"

* Retains the declaration of wildfly.jar as a compile-time
dependency in the hadoop-aws POM. This means that unless
explicitly excluded, applications importing that published
maven artifact will, transitively, add the specified
wildfly JAR into their classpath for compilation/testing/
distribution.

This is done for packaging and to offer that optional
speedup. It is not mandatory: applications importing
the hadoop-aws POM can exclude it if they choose.

Change-Id: I7ed3e5948d1e10ce21276b3508871709347e113d
 2020-04-20 14:42:36 +01:00
Sammi Chen
9c81b17153 HADOOP-16959. Resolve hadoop-cos dependency conflict. Contributed by Yang Yu. 
(cherry picked from commit 82ff7bc9ab)
 2020-04-20 18:57:49 +08:00
Surendra Singh Lilhore
a6c718fd0f HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 8e6227441a)
 2020-04-18 20:21:01 +05:30
Masatake Iwasaki
de5d43300a HADOOP-16972. Ignore AuthenticationFilterInitializer for KMSWebServer. (#1961) 
(cherry picked from commit ac40daece1)
 2020-04-18 06:40:18 +09:00
Jonathan Hung
49ae9b2137 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:59 -07:00
Mukund Thakur
96d7ceb39a
HADOOP-13873. log DNS addresses on s3a initialization. 
Contributed by Mukund Thakur.

If you set the log org.apache.hadoop.fs.s3a.impl.NetworkBinding
to DEBUG, then when the S3A bucket probe is made -the DNS address
of the S3 endpoint is calculated and printed.

This is useful to see if a large set of processes are all using
the same IP address from the pool of load balancers to which AWS
directs clients when an AWS S3 endpoint is resolved.

This can have implications for performance: if all clients
access the same load balancer performance may be suboptimal.

Note: if bucket probes are disabled, fs.s3a.bucket.probe = 0,
the DNS logging does not take place.

Change-Id: I21b3ac429dc0b543f03e357fdeb94c2d2a328dd8
 2020-04-17 14:20:54 +01:00
Stephen O'Donnell
aaad947c74 HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a)
 2020-04-16 20:19:00 -07:00
Vinayakumar B
b4ba9bed7c
HADOOP-16985. Handle release package related issues (#1957) 
(cherry picked from commit 4d24d99e85)
 2020-04-15 23:09:41 +05:30
Mukund Thakur
94da630cd2
HADOOP-16465 listLocatedStatus() optimisation (#1943) 
Contributed by Mukund Thakur

Optimize S3AFileSystem.listLocatedStatus() to perform list
operations directly and then fallback to head checks for files

Change-Id: Ia2c0fa6fcc5967c49b914b92f41135d07dab0464
 2020-04-15 17:04:55 +01:00