big-data/hadoop
big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
24,297 Commits 2 Branches 2 Tags
Commit Graph

5909 Commits

Author SHA1 Message Date
Inigo Goiri
813aafcee2 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 
(cherry picked from commit 6e04b00df1bf4f0a45571c9fc4361e4e8a05f7ee)
 2021-03-24 17:44:29 +08:00
Karthik Palanisamy
c96821ece8 HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 
(cherry picked from commit 735e85a1d2036585075b81d1fb2fd302e89eb363)
 2021-03-24 17:44:29 +08:00
sunlisheng
a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944e628345109b43cba2cd4d49ca8cc6b)
 2021-03-24 14:04:47 +08:00
Narges Shadab
f07bde90c9 HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 
(cherry picked from commit 115623a6ee4eb7391c7c26e1728fa153c0698803)
 2021-03-24 14:04:47 +08:00
Narges Shadab
3a355282c9 HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe90e3bb2ee783f956432db11e655b9e3)
 2021-03-24 14:04:47 +08:00
Narges Shadab
4abe6e6ac1 HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0604484ec3c48ac2acb32ea71a53d9a)
 2021-03-24 14:04:47 +08:00
Toshihiro Suzuki
e9b06d942c HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 
(cherry picked from commit c12ddbd1de40b32bbe2f6a3e484abf843d6d92ae)
 2021-03-24 14:04:47 +08:00
Viraj Jasani
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2796) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:51:21 -07:00
Ayush Saxena
76c40a52d7 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:13:52 +05:30
Ayush Saxena
b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f669b2e2a6654cfacff22a090c462effc)
 2021-03-23 15:34:48 +00:00
Ayush Saxena
968c95bfba HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit ae13a5ccbea10fe86481adbbff574c528e03c7f6)
 2021-03-23 15:34:36 +00:00
Ayush Saxena
d8314fffc0 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 
(cherry picked from commit c2a17659d1d48407edf437131783b653dbc87bc1)
 2021-03-22 12:38:09 +00:00
Ayush Saxena
793ab9acf3 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 12:35:11 +00:00
litao
bc2bdee76c HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 
(cherry picked from commit 343ce8a8c955dcf64ad453e441947e8761247a5b)
 2021-03-22 20:30:53 +09:00
Ayush Saxena
0b43c65816 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad2396f8b42d60031001a25d9d6fa32e)
 2021-03-19 18:50:31 +00:00
Viraj Jasani
17307ee88d HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:13:27 +09:00
Takanobu Asanuma
aec9d2cff7 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:15:53 +09:00
Hui Fei
34f7562abe HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:07:48 +08:00
tomscut
c498ef4b6d HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:00:25 -08:00
litao
852adff194 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:35:45 +09:00
Konstantin V Shvachko
56679e83bf HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:20:40 -08:00
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5d0024ab052429ef2f0c0b20e1ea258)
 2021-02-24 11:04:52 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2fa83ad4119954b32a3a8daac9d08134)
 2021-02-22 18:14:18 +08:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8bf96cf5a1ec75134fb67fdeb955e5b6)
 2021-02-22 15:24:37 +08:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a978e1b5220a961813d0e0c834010553)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd27c69b86b837e8da320bad0031f7895)
 2021-02-15 17:32:14 +00:00
fermi
ae80fc2477 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c8932c54ced1f0d4b892a54d22d7cfaf3)
 2021-02-05 11:05:18 +08:00
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc02af81e89e9fb96691f6cdf19ceff5)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc73cac4764149fcc9c7d94c1541f89c)
 2021-02-03 08:58:10 -06:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e204d105fb05807d33387a048e9ddb762)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939332c2b17a0f470ac650ab55c6c0f64)
 2021-01-28 18:19:21 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a8470975e67cce0e3e4d8eb8c7b6e7b6992f)
 2021-01-21 10:09:17 +09:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
 2021-01-04 11:28:26 -08:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234256e342fd9b5ef6fe4455471009693)
 2020-12-17 17:50:47 +00:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
S O'Donnell
1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c9c3d54f618e802fddacbafbe828211)
 2020-12-11 21:05:00 +00:00
Ahmed Hussein
16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50672ebdd7f12b9a2235aec37b2a07cf)
 2020-12-10 13:50:32 -08:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Hui Fei
cb2dce30d4 HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 10:40:14 +08:00
crossfire
edd9b659ca HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d5765c98d41c78736124b7b7f078aa2)
 2020-12-07 15:50:38 -08:00
Jim Brennan
6a5864ee4a HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 2020-12-07 21:19:36 +00:00
Ahmed Hussein
2249edb001 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69044a0770390f674d93f57e4bab3880)
 2020-12-02 21:40:17 +00:00
Ahmed Hussein
0f18a19d97 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7f17f095548b4a4485e7b7218af74be)
 2020-12-02 20:12:31 +00:00
Ahmed Hussein
a660714387 HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82fb0bc6f96301c7ca74a32018b60c49e)
 2020-12-01 21:16:49 +00:00
Takanobu Asanuma
d78bf82722 HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d9381d13329d25bb4f367e5378b6b5b)
 2020-11-13 11:05:20 +09:00