big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,687 Commits 2 Branches 2 Tags 264 MiB
176bd88890
Commit Graph

984 Commits

Author SHA1 Message Date
He Xiaoqiao
4a0b7f7ebe
HDFS-15384. Document getLocatedBlocks(String src, long start) of DFSClient only return partial blocks. Contributed by Yang Yun. 2021-03-08 12:18:46 +08:00
zhuqi
1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. (#2721). Contributed by Qi Zhu 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-02 13:16:11 +08:00
Ayush Saxena
ff84a57483 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 2021-02-20 19:48:00 +05:30
Akira Ajisaka
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
Jim Brennan
c4918fb298 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 21:20:30 +00:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
sunlisheng
f8769e0f4b HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 2021-01-28 10:10:39 +08:00
Yiqun Lin
da1ea2530f HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 2020-12-07 18:52:12 +08:00
Siyao Meng
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472) 2020-11-25 11:01:04 -08:00
He Xiaoqiao
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2020-11-23 11:26:52 +08:00
Akira Ajisaka
b57f04cd5b
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 2020-11-17 10:56:56 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
Siyao Meng
fbd2220167
HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448) 2020-11-10 12:13:15 -08:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
Takanobu Asanuma
30f06e0c74
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 2020-10-20 19:08:59 +09:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Ayush Saxena
cc57eebe45
HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2020-10-16 10:36:24 +05:30
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
bshashikant
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. (#2355) 2020-10-07 10:34:32 +05:30
Doroszlai, Attila
1cfe5916e2 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-06 14:58:42 -07:00
Siyao Meng
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir (#2352) 2020-10-05 05:02:00 -07:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00
Uma Maheswara Rao G
3ccc962b99
HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-24 07:07:48 -07:00
huangtianhua
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
Uma Maheswara Rao G
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:18:10 +05:30
Takanobu Asanuma
5c5b2ed7c7 HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui. 2020-09-16 01:36:46 +09:00
Siyao Meng
6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258) 2020-09-14 13:31:34 -07:00
imbajin
89428f142f
HDFS-15551. Tiny Improve for DeadNode detector (#2265) 
Contributed by imbajin.

Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-09-11 15:44:03 +08:00
1996fanrui
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache (#2241) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-09-10 22:30:52 -07:00
Siyao Meng
43c52d6495
HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir (#2295) 
Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
 2020-09-10 15:55:23 +05:30
He Xiaoqiao
1d6d0d8207 HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun. 
Co-authored-by: Chen Zhang <chzhang1987@gmail.com>

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-09-09 20:09:13 +08:00
Uma Maheswara Rao G
ac7d4623ae
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275) Contributed by Uma Maheswara Rao G. 2020-09-07 11:36:13 -07:00
Uma Maheswara Rao G
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 2020-08-19 09:30:41 -07:00
Vivek Ratnavel Subramanian
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots (#2212) 2020-08-13 22:36:15 +05:30
Siyao Meng
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory (#2176) 2020-08-11 08:52:16 -07:00
Akira Ajisaka
32895f4f7e
HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 2020-08-11 13:49:56 +09:00
Takanobu Asanuma
1d5ccc790b
HDFS-15512. Remove smallBufferSize in DFSClient. (#2191) 2020-08-06 22:47:12 +05:30
Akira Ajisaka
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors (#2098) 2020-08-03 10:46:51 +09:00
bshashikant
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181) 2020-08-02 21:59:56 +05:30
bshashikant
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166) 2020-07-29 21:33:25 +05:30
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Steve Loughran
b9fa5e0182
HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:30:02 +01:00
He Xiaoqiao
4f26454a7d HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang. 2020-07-08 00:02:39 +08:00
Yiqun Lin
9cbd76cc77 HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
Eric Yang
81d8a887b0
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-06-15 10:56:16 +09:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G
93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 2020-06-10 15:00:02 -07:00