big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,702 Commits 2 Branches 2 Tags 264 MiB
cbb3ba135c
Commit Graph

9123 Commits

Author SHA1 Message Date
S O'Donnell
e3ef326663 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:04:52 +00:00
Ayush Saxena
6ff5da931f HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:12:42 +05:30
Wang Yuxuan
52a46f064a HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
 2021-02-22 18:14:18 +08:00
jianghuazhu
ee0e32abdc HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
 2021-02-22 15:24:37 +08:00
Takanobu Asanuma
1125e3f0c0 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 
(cherry picked from commit 25af8901a9)
 2021-02-19 14:47:16 +09:00
Hui Fei
a9d3500894 HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:14:14 +08:00
Akira Ajisaka
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b1733)
 2021-02-16 14:52:31 +09:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:32:14 +00:00
Steve Loughran
98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
Jim Brennan
62389a5a04 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 22:16:36 +00:00
fermi
ae80fc2477 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:05:18 +08:00
Hui Fei
164a4e63e0 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 09:58:53 +08:00
Kihwal Lee
96773ec81b HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 
(cherry picked from commit 182623d2bc)
 2021-02-03 08:58:10 -06:00
crossfire
e0f8462b39 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 09:07:23 +00:00
He Xiaoqiao
f97709beaa
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:52:08 +08:00
He Xiaoqiao
b431c6924a
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:34:13 +08:00
Jim Brennan
37d8ad7ade [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:19:21 +00:00
Jim Brennan
2c70705575 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 16:51:40 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:09:17 +09:00
He Xiaoqiao
26cd02fb29
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-19 23:29:45 +08:00
Masatake Iwasaki
89452682a5
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:20:49 +09:00
touchida
75f34a156f
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:25:10 +09:00
He Xiaoqiao
e95ee67632
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:10 +08:00
Ahmed Hussein
18e2835766 HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 
(cherry picked from commit 77435a025e)
 2021-01-08 19:29:14 +00:00
Wei-Chiu Chuang
94c126cc9e HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
 2021-01-04 20:56:18 -08:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0)
 2021-01-04 11:28:26 -08:00
Liang-Chi Hsieh
060594259c HDFS-15690. Add lz4-java as test dependency (#2481) 2020-12-29 13:36:04 -08:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 17:50:47 +00:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
S O'Donnell
1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c)
 2020-12-11 21:05:00 +00:00
Ahmed Hussein
16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
 2020-12-10 13:50:32 -08:00
Jim Brennan
27e455a1a0 HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 20:57:43 +00:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Ahmed Hussein
e02b179c4c HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a2172)
 2020-12-08 19:28:44 +00:00
Hui Fei
cb2dce30d4 HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 10:40:14 +08:00
crossfire
edd9b659ca HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
 2020-12-07 15:50:38 -08:00
Jim Brennan
6a5864ee4a HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 2020-12-07 21:19:36 +00:00
Chao Sun
4628647067 HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:43:25 -08:00
Ahmed Hussein
a7dbd3b6f9 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040)
 2020-12-03 15:15:18 -08:00
Ahmed Hussein
2249edb001 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69)
 2020-12-02 21:40:17 +00:00
Ahmed Hussein
0f18a19d97 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7)
 2020-12-02 20:12:31 +00:00
Ahmed Hussein
a660714387 HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82f)
 2020-12-01 21:16:49 +00:00
Akira Ajisaka
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287)
 2020-11-26 11:34:39 +09:00
Akira Ajisaka
be108159b1
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 
(cherry picked from commit b57f04cd5b)
 2020-11-17 11:01:23 +09:00
Takanobu Asanuma
d78bf82722 HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d)
 2020-11-13 11:05:20 +09:00
Yizhong Zhang
12bfd9194a HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794)
 2020-11-10 09:14:12 -08:00
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 14:52:05 -08:00
Akira Ajisaka
8e0922edb2 HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 
(cherry picked from commit b3ba74d72d)
 2020-10-28 19:14:19 +09:00
Ahmed Hussein
d4129c6794
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4)
 2020-10-27 20:49:15 +09:00
Ahmed Hussein
1429c65533
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d332)
 2020-10-27 16:58:04 +09:00
Hui Fei
b8cc09d7c7 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:25:03 +08:00
Wei-Chiu Chuang
c8eb86ee3d HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
 2020-10-23 12:06:23 -07:00
Inigo Goiri
c55d639e8f HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a2)
 2020-10-23 09:48:23 -07:00
Kihwal Lee
02709cb054 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 19:41:38 -05:00
Kihwal Lee
cf932a7e2d HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-22 09:55:28 -05:00
Takanobu Asanuma
0bb1f0df27 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:12:26 +09:00
Ayush Saxena
54c40cbf49
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383) 
Contributed by Gabor Bota.
 2020-10-17 01:34:01 +05:30
Kihwal Lee
433434d509 HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:16:47 -05:00
Kihwal Lee
41a3c9bc95 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d)
 2020-10-14 17:28:02 -05:00
Mukund Thakur
351b1498d3
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:32:35 +01:00
Ayush Saxena
2a043b987c
HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 
(cherry picked from commit 7a6265ac42)
 2020-10-13 18:29:54 +09:00
Akira Ajisaka
2e73871cab
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 
(cherry picked from commit 69ef9b1ee8)
 2020-10-13 17:43:29 +09:00
Konstantin V Shvachko
b6423d2780 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:38:42 -07:00
Akira Ajisaka
800b1ed1c2 Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-12 11:59:17 -07:00
Steve Loughran
7cf5bdeec0 Revert "HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375)" 
This reverts commit 263b7d5dfc.
 2020-10-12 10:45:18 +01:00
Akira Ajisaka
263b7d5dfc HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 10:43:26 +01:00
Doroszlai, Attila
13e0c5f6e0 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-08 23:44:56 -07:00
Akira Ajisaka
89314a7bae
HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360) 
* Support getHAServiceState in DFSRouter

(cherry picked from commit 074f0d46af)
 2020-10-07 13:40:31 +09:00
Hui Fei
ed19f63998
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.

(cherry picked from commit 474fa80bfb)
 2020-09-23 15:39:51 +09:00
Kihwal Lee
7eb91ac1b2 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 
(cherry picked from commit dfc2682213)
 2020-09-22 10:55:26 -05:00
S O'Donnell
5f321df0a0 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-22 12:00:02 +01:00
Uma Maheswara Rao G
2ce5846bfa HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:23:35 +05:30
Uma Maheswara Rao G
1fc1b34633 HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275) Contributed by Uma Maheswara Rao G. 2020-09-18 15:23:12 +05:30
Uma Maheswara Rao G
2d9c5395ef HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit e4cb0d3514)
 2020-09-16 23:01:03 -07:00
hemanthboyina
94e5c5257f HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 09:40:36 +05:30
Akira Ajisaka
0731c8b5d0
HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267) 
(cherry picked from commit c78d18023d)
 2020-09-14 11:36:26 +09:00
Uma Maheswara Rao G
1195dac55e HDFS-15529: getChildFilesystems should include fallback fs as well (#2234). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b3660d0147)
 2020-09-12 20:48:59 -07:00
Uma Maheswara Rao G
bfa145dd7c HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d2779de3f5)
 2020-09-12 20:43:34 -07:00
zz
2d5ca83078 HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185). Contributed by Zhenzhao Wang. 
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
(cherry picked from commit 12a316cdf9)
 2020-09-12 20:42:06 -07:00
Mingliang Liu
4eccdd950f
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell 2020-09-12 01:50:28 -07:00
hemanthboyina
a2bc0dcd21 HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui. 
(cherry picked from commit 2a6fdcedeb)
 2020-09-11 18:22:49 +05:30
Steve Loughran
1b9109d237
HDFS-15471. TestHDFSContractMultipartUploader failing (#2252) 
Contributed by Steve Loughran
(Was: broken by Steve Loughran)

Change-Id: If6a82706f3ea6d802bc6da03c2a2ca734e30388f
 2020-08-28 15:47:06 +01:00
Stephen O'Donnell
a9ce6001ea HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2ffe00fc46)
 2020-08-26 23:07:27 -07:00
Ayush Saxena
9eaa3520e6 HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu. 
(cherry picked from commit 0fe49036e5)
 2020-08-26 23:07:22 -07:00
hemanthboyina
ac0a377779 HDFS-15536. RBF: Clear Quota in Router was not consistent. 
(cherry picked from commit 6e618b6a7e)
 2020-08-26 13:04:47 +05:30
Uma Maheswara Rao G
ba0eca6a2c HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dd013f2fdf)
 2020-08-25 12:00:52 -07:00
S O'Donnell
97d6058aa6 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 15:37:52 +01:00
hemanthboyina
60d167d080 HDFS-14504. Rename with Snapshots does not honor quota limit. 
(cherry picked from commit 83fd15b412)
 2020-08-23 16:12:36 +05:30
Chen Liang
b1292bdaef HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d)
 2020-08-20 15:27:52 -07:00
Uma Maheswara Rao G
99b120a06e HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test
 2020-08-13 14:32:19 +01:00
S O'Donnell
42154f04e3 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 
(cherry picked from commit 10716040a8)
 2020-08-12 09:33:14 +01:00
Gautham B A
b4a105a209
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3)
 2020-08-11 16:35:41 +09:00
Ayush Saxena
2943e6650f HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:20:42 +05:30
Mingliang Liu
1cd1f978da
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 20:49:50 -07:00
hemanthboyina
b5d712251c HDFS-15503. File and directory permissions are not able to be modified from WebUI. 
(cherry picked from commit 82f3ffcd64)
 2020-08-03 23:43:29 +05:30
Uma Maheswara Rao G
4fe491d10e HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:31:37 -07:00
Uma Maheswara Rao G
ae8261c671 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:31:15 -07:00
Uma Maheswara Rao G
10f8010519 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:30:54 -07:00
Uma Maheswara Rao G
aea1a8e2bd HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:30:30 -07:00
Uma Maheswara Rao G
35fe6fd54f HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5)
 2020-07-31 01:29:46 -07:00
Yuan
e1ac16a318
HDFS-14950. fix missing libhdfspp lib in dist-package (#1947) 
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit e756fe3590)
 2020-07-31 16:51:05 +09:00
Ayush Saxena
e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Chen Liang
c8c40be761 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 15:22:34 -07:00
Ahmed Hussein
9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Ahmed Hussein
5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 09:57:36 -05:00
Shashikant Banerjee
292e22578a HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 
(cherry picked from commit 82343790ee)
 2020-07-15 13:13:27 +01:00
Steve Loughran
a51d72f0c6 HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:32:04 +01:00
He Xiaoqiao
e7e7a6d503 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:31:50 +08:00
Uma Maheswara Rao G
f85ce2570e HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:26:04 -07:00
Shanyu Zhao
10c9df1d0a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 11:44:42 +08:00
Ayush Saxena
b8d9218189 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:25:29 +05:30
Akira Ajisaka
3e9422d1c7
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:04:47 +09:00
Virajith Jalaparti
ea97fe250c HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:22:50 -07:00
Uma Maheswara Rao G
81e33d22a0 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 15:42:36 -07:00
Uma Maheswara Rao G
29a8ee4be6 HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 15:42:14 -07:00
Uma Maheswara Rao G
5f67c3f3ca HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 15:41:48 -07:00
Uma Maheswara Rao G
3cddd0be29 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 15:39:38 -07:00
Stephen O'Donnell
7613191fcd HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 630ac9e7e5fdbc3ce86476cf0167255ab9b0470a)
 2020-06-18 06:45:28 -07:00
Wei-Chiu Chuang
10880dc396 Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." 
This reverts commit 0b9e5ea592.
 2020-06-18 06:45:28 -07:00
S O'Donnell
3dc9db3aed HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 
(cherry picked from commit 123777823e)
 2020-06-18 12:29:12 +01:00
Uma Maheswara Rao G
120ee793fc HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def95)
 2020-06-16 20:02:44 -07:00
Ayush Saxena
bee2846bee HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7)
 2020-06-16 16:54:40 -07:00
Uma Maheswara Rao G
418580446b HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 76fa0222f0)
 2020-06-16 16:54:01 -07:00
Uma Maheswara Rao G
0b5e202614 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
 2020-06-16 16:53:38 -07:00
Uma Maheswara Rao G
8e71e85af7 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b)
 2020-06-16 16:53:10 -07:00
Abhishek Das
5b248de42d HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec74453)
 2020-06-16 16:52:29 -07:00
Uma Maheswara Rao G
544996c857 HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d9)
 2020-06-16 16:50:57 -07:00
Stephen O'Donnell
0b9e5ea592 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 730a39d138)
 2020-06-16 16:02:07 -07:00
Eric Yang
d73cdb1c86
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0)
 2020-06-15 11:12:08 +09:00
Takanobu Asanuma
3b8418250f HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077)
 2020-06-15 09:17:40 +09:00
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Ayush Saxena
043628dcf1 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:09:34 +05:30
Ayush Saxena
2d4faa39e8 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:47:24 +05:30
Toshihiro Suzuki
ec8f3714e0 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)
 2020-06-05 11:17:13 +01:00
Wanqiang Ji
9b84a637b7
HADOOP-17055. Remove residual code of Ozone (#2039) 
(cherry picked from commit d9838f2d42)
 2020-05-29 16:50:10 +09:00
Ayush Saxena
6b7040a1cb HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:04:22 +05:30
Ayush Saxena
718c8a5868 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:06:19 +05:30
Takanobu Asanuma
c44d8b8bdf HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:33:39 +09:00
Chen Liang
e452163a06
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 
(cherry picked from commit 7bb902bc0d)
 2020-05-19 09:52:25 +09:00
He Xiaoqiao
eb045ea056 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 09:59:12 -07:00
Ayush Saxena
3915d1afc7 HDFS-15356. Unify configuration dfs.ha.allow.stale.reads to DFSConfigKeys. Contributed by Xiaoqiao He. 
(cherry picked from commit 178336f8a8)
 2020-05-18 09:58:59 -07:00
Wei-Chiu Chuang
240cba7e6d HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
(cherry picked from commit 2abcf7762a)
 2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang
289f8acc64 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit ad9a6a0ee3.
 2020-05-18 09:41:03 -07:00
Wei-Chiu Chuang
53d22fdb88 Revert "HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He." 
This reverts commit acae31aa28.
 2020-05-18 09:39:57 -07:00
Wei-Chiu Chuang
032ccba67c HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 
(cherry picked from commit 50caba1a92)
 2020-05-18 08:42:20 -07:00
He Xiaoqiao
acae31aa28 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 07:10:27 -07:00
pustota2009
ad9a6a0ee3 HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

(cherry picked from commit 86e6aa8eec)
 2020-05-18 07:05:03 -07:00
Surendra Singh Lilhore
7cef7109de YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:17:13 +05:30
Ayush Saxena
74887ab2ad YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:09 +05:30
Akira Ajisaka
ee58c112d0
Revert "SPNEGO TLS verification" 
This reverts commit 0f27c04c23.
 2020-05-13 17:14:51 +09:00
S O'Donnell
433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena
ef41229f70 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:43:37 +05:30
Babneet Singh
d1e5e393c3 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
 2020-05-11 10:45:57 -07:00
Ayush Saxena
61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Ayush Saxena
6ccb1cd5f1 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Ayush Saxena
817814105a HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:06 +05:30
Uma Maheswara Rao G
edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-06 15:13:33 -07:00
Ayush Saxena
4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang
78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko
8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:45:18 -07:00
Ayush Saxena
df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka
e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
Eric Yang
0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:51:12 +09:00
Akira Ajisaka
dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell
a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9a)
 2020-04-29 16:01:48 +01:00
Aryan Gupta
08dbe32122 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 
(cherry picked from commit 816042e62b)
 2020-04-29 11:16:56 +05:30
Akira Ajisaka
d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:18:25 +09:00
Mingliang Liu
1db1992b31
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:44:01 -07:00
Dhiraj
1c19107ce8
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:06:08 -07:00
Wei-Chiu Chuang
d6c48f8100 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
 2020-04-24 11:54:11 -07:00
Mingliang Liu
47b330dc34
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:41:04 -07:00
Akira Ajisaka
5459dd64fd
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 
(cherry picked from commit e069a06137)
 2020-04-21 10:27:27 +09:00
Surendra Singh Lilhore
a6c718fd0f HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 8e6227441a)
 2020-04-18 20:21:01 +05:30
Stephen O'Donnell
aaad947c74 HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a)
 2020-04-16 20:19:00 -07:00
Wei-Chiu Chuang
749a5b81da HDFS-15269. NameNode should check the authorization API version only … (#1945) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 061afcdf30)
 2020-04-09 09:45:05 -07:00
Akira Ajisaka
d501c6a43d
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 
(cherry picked from commit 1695d8d59c)
 2020-04-04 00:16:52 +09:00
Inigo Goiri
f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena
696a663cd0 HDFS-15193. Improving the error message for missing dfs.namenode.rpc-address.$NAMESERVICE. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Chen Liang
f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00
Ayush Saxena
cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Inigo Goiri
679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
Wei-Chiu Chuang
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909) 2020-03-25 16:03:26 -07:00
Inigo Goiri
cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Arpit Agarwal
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. 
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
 2020-03-25 10:28:30 -07:00
Toshihiro Suzuki
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Isa Hekmatizadeh
d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 2020-03-24 23:17:22 +05:30
Ayush Saxena
ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Karthik Palanisamy
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
He Xiaoqiao
5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 22:41:31 -07:00
Akira Ajisaka
f59f6891c8
HDFS-15232. Fix libhdfspp test failures with GCC 7. (#1906) 2020-03-24 11:36:14 +09:00
He Xiaoqiao
af64ce2f4a HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 12:43:51 -07:00
Ayush Saxena
62c8015683 HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena. 2020-03-22 11:44:18 +05:30
Takanobu Asanuma
b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
Ayush Saxena
f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena
bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Ayush Saxena
654db35fa2 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
 2020-03-18 13:27:13 +00:00
Surendra Singh Lilhore
1d772dc542 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:44:32 +05:30
Ayush Saxena
74780c22eb HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:30:39 +05:30
Ayush Saxena
8a7c54995a HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina. 2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang
7b637a3341 HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun. 2020-03-13 19:01:23 -07:00
He Xiaoqiao
e9955bb8ff HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829) 
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
 2020-03-13 11:29:44 -07:00
Ayush Saxena
fb1d7635ae HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang. 2020-03-13 21:10:03 +05:30
S O'Donnell
25497799cb HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang
20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri
0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore
f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Steve Loughran
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface 
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
 2020-03-09 14:51:16 +00:00
Steve Loughran
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden 
Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
 2020-03-09 14:44:28 +00:00
Inigo Goiri
4062217189 HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena. 2020-03-06 10:14:53 -08:00
Inigo Goiri
3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Inigo Goiri
1d4d0fcbe1 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 2020-03-04 10:02:54 -08:00
Akshay Nehe
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871) 2020-03-04 11:07:31 +09:00
Xiaoyu Yao
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file 
Contributed by Xiaoyu Yao.

Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails

Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.

This is a convergence of a fix for S3 meeting an issue in HDFS.

S3: a HEAD against a file can cache a 404, 
 -you must not do any existence checks, including deleteOnExit(),
  until the file is written. 

Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct. 

HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.

This patch:

* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
 2020-03-02 13:22:00 +00:00
Ayush Saxena
1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Inigo Goiri
97b797c314 HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun. 2020-02-28 18:47:22 -08:00
Takanobu Asanuma
9a3a28018a HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina. 2020-02-29 10:30:26 +09:00
Surendra Singh Lilhore
fccfb02672 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:33:06 +05:30
Ayush Saxena
429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena
cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Istvan Fajth
57aa048516
HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 2020-02-27 07:48:14 -08:00
Surendra Singh Lilhore
2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena
033a3d7ff9 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:26 +05:30
Ayush Saxena
209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee
352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell
900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang
1c5d2f1fdc HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 2020-02-24 13:39:16 -08:00
Ayush Saxena
7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena
93b8f453b9 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:01 +05:30
Ayush Saxena
b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena
9eb7a8bdf8 HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi. 2020-02-23 14:25:39 +05:30
Ayush Saxena
ba9025c7cd HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun. 2020-02-22 14:27:26 +05:30
Konstantin V Shvachko
ecbcb058b8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 2020-02-21 17:53:37 -08:00
Inigo Goiri
ed70c115a8 HDFS-15172. Remove unnecessary deadNodeDetectInterval in DeadNodeDetector#checkDeadNodes(). Contributed by Lisheng Sun. 2020-02-21 16:36:30 -08:00
Konstantin V Shvachko
6f84269bcd HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 2020-02-21 10:51:14 -08:00
Masatake Iwasaki
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 2020-02-21 11:56:07 +09:00
Inigo Goiri
ec7507162c HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham. 2020-02-19 11:33:58 -08:00
Ayush Saxena
ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Ayush Saxena
84f7638840 HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri. 2020-02-18 00:43:33 +05:30
xiaoxiaopan118
439d935e1d
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 2020-02-17 15:55:10 +09:00
Surendra Singh Lilhore
810783d443 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-02-16 12:15:03 +05:30
Ayush Saxena
c75756fe13 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore
a98352ced1 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 16:57:41 +05:30
Akira Ajisaka
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router (#1832) 
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
 2020-02-13 10:06:07 +09:00
Ayush Saxena
f09710bbb8 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:35 +05:30
Ayush Saxena
3df0adaaea HDFS-15127. RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Contributed by Inigo Goiri 2020-02-12 19:41:04 +05:30