big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,916 Commits 2 Branches 2 Tags 264 MiB
0c25131ca4
Commit Graph

5202 Commits

Author SHA1 Message Date
Mingliang Liu
0c25131ca4
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:28:56 -07:00
Mingliang Liu
9f242c215e
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:11:30 -07:00
Ayush Saxena
a8610c15c4
HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 2020-06-07 13:29:10 -07:00
Uma Maheswara Rao G
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 2020-06-06 08:11:57 -07:00
Abhishek Das
e7dd02768b
HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 2020-06-05 14:56:51 -07:00
Mike
97c98ce531
HADOOP-14566. Add seek support for SFTP FileSystem. (#1999) 
Contributed by Mikhail Pryakhin
 2020-06-03 11:37:40 +01:00
Xiaoyu Yao
6288e15118 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 2020-06-02 11:53:32 -07:00
Uma Maheswara Rao G
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00
Dhiraj
9fe4c37c25
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 10:49:17 -07:00
S O'Donnell
19f26a020e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 2020-05-29 14:36:56 +01:00
S O'Donnell
d9e8046a1a HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 10:32:37 +01:00
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Uma Maheswara Rao G
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 2020-05-21 21:34:58 -07:00
Akira Ajisaka
52b21de1d8
HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027) 
* Removed javax.activation-api from dependency
 2020-05-22 11:19:11 +09:00
Kei Kori
1a3c6bb33b
HDFS-15353. Use sudo instead of su to allow nologin user for secure DataNode (#2018) 2020-05-21 10:07:23 +09:00
Abhishek Das
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 2020-05-18 22:27:12 -07:00
Akira Ajisaka
27601fc79e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 2020-05-18 15:36:20 +09:00
Uma Maheswara Rao G
ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 2020-05-14 17:29:35 -07:00
Mike
017d24e970
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists. 
Contributed by Mikhail Pryakhin.
 2020-05-14 18:28:00 +01:00
Joseph Smith
d60496e6c6
BytesWritable causes OOME when array size reaches Integer.MAX_VALUE. (#393) 2020-05-13 00:20:35 +05:30
Ayush Saxena
936bf09c37 HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 2020-05-12 21:54:54 +05:30
Elixir Kook
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007) 
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
 2020-05-12 10:50:04 -05:00
S O'Donnell
29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Akira Ajisaka
328eae9a14
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003) 2020-05-11 14:44:18 +09:00
Akira Ajisaka
c784ba370e
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 2020-05-08 17:18:23 +09:00
Mingliang Liu
e9e1ead089
HADOOP-17027. Add tests for reading fair call queue capacity weight configs. Contributed by Fengnan Li 2020-05-07 16:50:23 -07:00
Uma Maheswara Rao G
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 2020-05-04 17:55:40 -07:00
Mingliang Liu
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-04-30 14:15:28 -07:00
Ayush Saxena
6bdab3723e HADOOP-16957. NodeBase.normalize doesn't removing all trailing slashes. Contributed by Ayush Saxena. 2020-04-30 19:55:20 +05:30
S O'Donnell
9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
lfengnan
4202750040
HADOOP-17010. Add queue capacity support for FairCallQueue (#1977) 2020-04-28 16:14:55 -07:00
Mike
18d7dfbf35
HDFS-1820. FTPFileSystem attempts to close the outputstream even when it is not initialised. (#1952) 
Contributed by Mikhail Pryakhin.
 2020-04-27 14:43:51 +01:00
Dhiraj
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:04:32 -07:00
Ayush Saxena
ef9a6e775c HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-04-25 13:07:32 +05:30
Mingliang Liu
af85971a58
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 12:36:33 -07:00
Steve Loughran
42711081e3
HADOOP-16986. S3A to not need wildfly on the classpath. (#1948) 
HADOOP-16986. S3A to not need wildfly JAR on its classpath.

Contributed by Steve Loughran

This is a successor to HADOOP-16346, which enabled the S3A connector
to load the native openssl SSL libraries for better HTTPS performance.

That patch required wildfly.jar to be on the classpath. This
update:

* Makes wildfly.jar optional except in the special case that 
"fs.s3a.ssl.channel.mode" is set to "openssl"

* Retains the declaration of wildfly.jar as a compile-time
dependency in the hadoop-aws POM. This means that unless
explicitly excluded, applications importing that published
maven artifact will, transitively, add the specified
wildfly JAR into their classpath for compilation/testing/
distribution.

This is done for packaging and to offer that optional
speedup. It is not mandatory: applications importing
the hadoop-aws POM can exclude it if they choose.
 2020-04-20 14:32:13 +01:00
Ayush Saxena
79e03fb622 HADOOP-16971. TestFileContextResolveAfs#testFileContextResolveAfs creates dangling link and fails for subsequent runs. Contributed by Ctest. 2020-04-20 01:59:18 +05:30
belugabehr
eca05917d6
HADOOP-16951: Tidy Up Text and ByteWritables Classes. 
1. Remove superfluous code
2. Remove superfluous comments
3. Checkstyle fixes
4. Remove methods that simply call super.method()
5. Use Java 8 facilities to streamline code where applicable
6. Simplify and unify some of the constructs between the two classes
7. Expanding of the arrays be 1.5x instead of 2x per expansion.
 2020-04-17 08:16:12 -07:00
Ayush Saxena
3edbe8708a HADOOP-16958. NPE when hadoop.security.authorization is enabled but the input PolicyProvider for ZKFCRpcServer is NULL. Contributed by Ctest. 2020-04-14 02:13:42 +05:30
Ayush Saxena
2ea5adf2de HADOOP-16967. TestSequenceFile#testRecursiveSeqFileCreate fails in subsequent run. Contributed by Ctest. 2020-04-12 11:31:07 +05:30
Ayush Saxena
5746533cde HADOOP-16962. Making getBoolean log warning message for unrecognized value. Contributed by Ctest. 2020-04-09 23:30:28 +05:30
Ayush Saxena
1189af4746 HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena. 2020-04-08 15:55:14 +05:30
Takanobu Asanuma
b89d875f7b HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 2020-04-06 11:03:10 +09:00
Masatake Iwasaki
8f8be6b92a HADOOP-16647. Support OpenSSL 1.1.1 LTS. Contributed by Rakesh Radhakrishnan. 2020-04-04 07:20:51 +09:00
He Xiaoqiao
d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-01 16:36:46 -07:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Chao Sun
e3fbdcbc14 HADOOP-16912. Emit per priority RPC queue time and processing time from DecayRpcScheduler. Contributed by Fengnan Li. 2020-03-25 10:21:20 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Wei-Chiu Chuang
f197f05cff
HADOOP-16661. Support TLS 1.3 (#1880) 2020-03-16 10:56:30 -07:00
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface 
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
 2020-03-09 14:51:16 +00:00