big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,334 Commits 2 Branches 2 Tags 264 MiB
1148d3895f
Commit Graph

8307 Commits

Author SHA1 Message Date
Inigo Goiri
1148d3895f HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 
(cherry picked from commit 130f89e068)
 2021-04-02 14:06:52 +08:00
Ayush Saxena
5187bd37ae HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 
(cherry picked from commit 37d6582223)
 2021-04-02 14:06:52 +08:00
maobaolong
2b207ea402 HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 
(cherry picked from commit 95c96605b3)
 2021-04-02 14:06:52 +08:00
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6)
 2021-04-02 10:57:05 +09:00
Viraj Jasani
f707d8407a
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:35:44 -07:00
He Xiaoqiao
e5d7b22b8b
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:59:53 +08:00
Brahma Reddy Battula
13878fc06b HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 0665ce9930)
 2021-04-01 09:22:18 +05:30
He Xiaoqiao
ab451b8e42 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa1)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9d)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb81)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf42)
 2021-03-25 13:25:17 +00:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aa)
 2021-03-25 05:09:11 +08:00
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1)
 2021-03-24 17:44:29 +08:00
Ayush Saxena
580634882e HDFS-14546. Document block placement policies. Contributed by Amithsha. 
(cherry picked from commit 17ffcab5f6)
 2021-03-24 17:44:29 +08:00
Karthik Palanisamy
c96821ece8 HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 
(cherry picked from commit 735e85a1d2)
 2021-03-24 17:44:29 +08:00
sunlisheng
a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944)
 2021-03-24 14:04:47 +08:00
Ayush Saxena
94766fdb13 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 
(cherry picked from commit ff84a57483)
 2021-03-24 14:04:47 +08:00
sunlisheng
3ecd02ac78 HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 
(cherry picked from commit f8769e0f4b)
 2021-03-24 14:04:47 +08:00
Narges Shadab
f07bde90c9 HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 
(cherry picked from commit 115623a6ee)
 2021-03-24 14:04:47 +08:00
Narges Shadab
3a355282c9 HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe9)
 2021-03-24 14:04:47 +08:00
Narges Shadab
4abe6e6ac1 HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0)
 2021-03-24 14:04:47 +08:00
Toshihiro Suzuki
e9b06d942c HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 
(cherry picked from commit c12ddbd1de)
 2021-03-24 14:04:47 +08:00
Viraj Jasani
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2796) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:51:21 -07:00
Ayush Saxena
76c40a52d7 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:13:52 +05:30
Ayush Saxena
b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f66)
 2021-03-23 15:34:48 +00:00
Ayush Saxena
968c95bfba HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit ae13a5ccbe)
 2021-03-23 15:34:36 +00:00
Ayush Saxena
d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena
793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58)
 2021-03-22 12:35:11 +00:00
litao
bc2bdee76c HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 
(cherry picked from commit 343ce8a8c9)
 2021-03-22 20:30:53 +09:00
Ayush Saxena
0b43c65816 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad)
 2021-03-19 18:50:31 +00:00
Viraj Jasani
17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma
aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:15:53 +09:00
touchida
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 12:01:15 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:07:48 +08:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:00:25 -08:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:35:45 +09:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
24a0304159 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)
 2021-03-01 11:50:23 +00:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:04:52 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
 2021-02-22 18:14:18 +08:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
 2021-02-22 15:24:37 +08:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a9)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:32:14 +00:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00