big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,737 Commits 2 Branches 2 Tags 264 MiB
afe6613ee6
Commit Graph

7719 Commits

Author SHA1 Message Date
Giovanni Matteo Fumarola
99e201dfe2 HDFS-14105. Replace TreeSet in NamenodeFsck with HashSet. Contributed by Beluga Behr. 2018-11-30 11:07:11 -08:00
Giovanni Matteo Fumarola
b09cfad432 HDFS-14106. Refactor NamenodeFsck#copyBlock. Contributed by Beluga Behr. 2018-11-30 10:47:59 -08:00
Yiqun Lin
0e36e935d9 HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. 2018-11-30 11:31:34 +08:00
Tsz Wo Nicholas Sze
0081b02e35 HDFS-14112. Avoid recursive call to external authorizer for getContentSummary. 2018-11-29 13:55:21 -08:00
Brahma Reddy Battula
f534736867 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 2018-11-30 00:18:27 +05:30
Ewan Higgs
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.
 2018-11-29 15:12:17 +00:00
Giovanni Matteo Fumarola
300f560fcc HDFS-14102. Performance improvement in BlockPlacementPolicyDefault. Contributed by Beluga Behr. 2018-11-28 11:33:22 -08:00
Giovanni Matteo Fumarola
4ca3a6b21a HDFS-14108. Performance improvement in BlockManager Data Structures. Contributed by Beluga Behr. 2018-11-28 11:25:47 -08:00
Vinayakumar B
b098281454 HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B. 2018-11-26 18:22:32 +05:30
Brahma Reddy Battula
892b33e054 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 2018-11-22 00:13:56 +05:30
Akira Ajisaka
1a00b4e325
Add 2.9.2 release notes and changes documents. 2018-11-20 14:25:18 +09:00
Vinayakumar B
68d5dfdc78 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 2018-11-16 18:05:10 +05:30
Vinayakumar B
8571507efa HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 2018-11-16 17:26:36 +05:30
Arpit Agarwal
1e15c7e855 HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel. 2018-11-15 12:42:31 -08:00
Inigo Goiri
460a94a10f HDFS-14045. Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN. Contributed by Jiandan Yang. 2018-11-15 10:58:57 -08:00
Inigo Goiri
993c2140cc HDFS-14054. TestLeaseRecovery2: testHardLeaseRecoveryAfterNameNodeRestart2 and testHardLeaseRecoveryWithRenameAfterNameNodeRestart are flaky. Contributed by Zsolt Venczel. 2018-11-15 09:29:14 -08:00
Xiao Chen
9da6054ca4 Revert "HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel." 
This reverts commit 7dc79a8b5b.
 2018-11-13 12:44:25 -08:00
Brahma Reddy Battula
e7b63baca1 HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better extensibility. Contributed by CR Hota 2018-11-13 12:45:44 +05:30
Arpit Agarwal
b6d4e19f34 HDFS-14065. Failed Storage Locations shows nothing in the Datanode Volume Failures. Contributed by Ayush Saxena. 2018-11-12 15:31:42 -08:00
Kitti Nanasi
8d99648c20 HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-11-08 10:01:19 -08:00
Inigo Goiri
887244de4a HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 2018-11-06 11:18:15 -08:00
Inigo Goiri
f0ce072934 HDFS-14051. Refactor NameNodeHttpServer#initWebHdfs to specify local keytab. Contributed by CR Hota. 2018-11-06 10:05:58 -08:00
Akira Ajisaka
6430c9890a
HADOOP-15904. [JDK 11] Javadoc build failed due to bad use of '>' in hadoop-hdfs. 2018-11-06 18:42:38 +09:00
Xiao Chen
ffc9c50e07 HDFS-14053. Provide ability for NN to re-replicate based on topology changes. Contributed by Hrishikesh Gadre. 2018-11-05 21:38:39 -08:00
Inigo Goiri
f3296501e0 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 2018-11-05 16:48:37 -08:00
Giovanni Matteo Fumarola
f3f5e7ad00 HDFS-14042. Fix NPE when PROVIDED storage is missing. Contributed by Virajith Jalaparti. 2018-11-05 11:02:31 -08:00
Siyao Meng
8fe85af63b HDFS-13996. Make HttpFS' ACLs RegEx configurable. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-11-01 17:36:50 -07:00
Pranay Singh
d98b881e9a HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-11-01 17:26:00 -07:00
Akira Ajisaka
fac9f91b29
HDFS-13942. [JDK10] Fix javadoc errors in hadoop-hdfs module. Contributed by Dinesh Chitlangia. 2018-10-31 14:43:58 +09:00
Xiao Chen
db7e636824 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 2018-10-29 19:06:15 -07:00
Wei-Chiu Chuang
63e7134df6 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fb2b72e6fc.
 2018-10-28 17:04:59 -07:00
Anu Engineer
4f10d7e23f HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.
 2018-10-26 16:41:23 -07:00
Inigo Goiri
1851d06eb3 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 2018-10-26 09:29:12 -07:00
Arpit Agarwal
9b899f1ebd HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 2018-10-25 15:14:13 -07:00
Wei-Chiu Chuang
fb2b72e6fc HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 2018-10-25 09:33:31 -07:00
Ajay Kumar
97bd49fc36 HDFS-14026. Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional. Contributed by Arpit Agarwal. 2018-10-24 21:55:42 -07:00
Inigo Goiri
936fc3f3c2 HDFS-14025. TestPendingReconstruction.testPendingAndInvalidate fails. Contributed by Ayush Saxena. 2018-10-24 16:18:43 -07:00
Inigo Goiri
c1874046e2 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 2018-10-24 09:20:38 -07:00
Chen Liang
635786a511 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2018-10-23 14:53:45 -07:00
Inigo Goiri
efdfe679d6 HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 2018-10-23 14:28:37 -07:00
Arpit Agarwal
c58811c77d HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 2018-10-22 14:44:28 -07:00
Inigo Goiri
f069d38c8d HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B. 2018-10-19 17:16:04 -07:00
Inigo Goiri
88cce32551 HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin. 2018-10-19 10:38:04 -07:00
Inigo Goiri
8b64fbab1a HDFS-13994. Improve DataNode BlockSender waitForMinLength. Contributed by BELUGA BEHR. 2018-10-19 10:36:03 -07:00
Inigo Goiri
be1cffb085 HDFS-14002. TestLayoutVersion#testNameNodeFeatureMinimumCompatibleLayoutVersions fails. Contributed by Takanobu Asanuma. 2018-10-18 15:25:53 -07:00
Inigo Goiri
0e56c883cd HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 2018-10-18 10:53:30 -07:00
Xiao Chen
533138718c HDFS-13662. TestBlockReaderLocal#testStatisticsForErasureCodingRead is flaky 2018-10-16 19:33:55 -07:00
Inigo Goiri
f90c64e624 HDFS-13993. TestDataNodeVolumeFailure#testTolerateVolumeFailuresAfterAddingMoreVolumes is flaky. Contributed by Ayush Saxena. 2018-10-16 11:22:57 -07:00
Xiao Chen
b6fc72a025 HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 2018-10-15 10:51:55 -07:00
Vinayakumar B
b60ca37914 Fix potential FSImage corruption. Contributed by Daryn Sharp. 2018-10-15 15:48:26 +05:30
Vinayakumar B
6e0e6daaf3 HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena. 2018-10-12 17:34:10 +05:30
Vinayakumar B
de44e1064f HDFS-13156. HDFS Block Placement Policy - Client Local Rack. Contributed by Ayush Saxena. 2018-10-12 17:27:23 +05:30
Shweta
c0118105b7 HDFS-13973. getErasureCodingPolicy should log path in audit event. Contributed by Shweta. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-11 20:39:41 -07:00
Inigo Goiri
3532aa3886 HDFS-13968. BlockReceiver Array-Based Queue. Contributed by BELUGA BEHR. 2018-10-11 13:49:39 -07:00
Steve Loughran
2addebb94f
HDFS-13951. HDFS DelegationTokenFetcher can't print non-HDFS tokens in a tokenfile. 
Contributed by Steve Loughran.
 2018-10-11 20:03:56 +01:00
Hrishikesh Gadre
bf3d591f0c HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:23:07 -07:00
Weiwei Yang
3ead525c71 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:11:10 -07:00
Kitti Nanasi
10185d9a77 HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-10 08:55:32 -07:00
Nanda kumar
b39b802422 HDFS-13949. Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml. Contributed by Toshihiro Suzuki. 2018-10-10 12:29:07 +05:30
Akira Ajisaka
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter. 2018-10-10 10:16:57 +09:00
Inigo Goiri
605622c87b HDFS-11396. TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out. Contributed by Ayush Saxena. 2018-10-09 17:03:21 -07:00
Hrishikesh Gadre
08bb6c49a5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-08 20:31:57 -07:00
Yiqun Lin
1043795f7f HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. 2018-10-09 10:33:13 +08:00
Inigo Goiri
73c660b43f HDFS-13958. Miscellaneous Improvements for FsVolumeSpi. Contributed by BELUGA BEHR. 2018-10-05 09:32:19 -07:00
Yiqun Lin
619e490333 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. 2018-10-05 09:55:08 +08:00
Adam Antal
cc2babc1f7 HDFS-13950. ACL documentation update to indicate that ACL entries are capped by 32. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-04 15:22:44 -07:00
Inigo Goiri
1dc0adfac0 HDFS-13947. Review of DirectoryScanner Class. Contributed by BELUGA BEHR. 2018-10-03 11:19:57 -07:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Yiqun Lin
5689355783 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-02 09:43:14 +08:00
Takanobu Asanuma
e2113500df HDFS-8196. Post enabled Erasure Coding Policies on NameNode UI. Contributed by Kitti Nanasi and Kai Sasaki. 2018-09-29 00:53:35 +09:00
Akira Ajisaka
f9c0221623
HDFS-13932. [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions. Contributed by Andrew Purtell. 2018-09-27 11:44:42 +09:00
Brahma Reddy Battula
e0ff8e2c10 HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 2018-09-26 23:44:16 +05:30
Inigo Goiri
6275b42870 HDFS-13927. Addendum: Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-26 09:50:09 -07:00
Inigo Goiri
4758b4b6da HDFS-13927. Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-21 15:32:28 -07:00
Anu Engineer
6fc293fece HDFS-13892. Disk Balancer: Make execute command documentation better. 
Contributed by  Ranith Sardar.
 2018-09-19 20:48:41 -07:00
Arpit Agarwal
2eb597b151 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 2018-09-19 13:48:27 -07:00
Chen Liang
a30b4f9e71 HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 13:22:37 -07:00
Inigo Goiri
090272d7de HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 2018-09-19 13:00:30 -07:00
Inigo Goiri
56e0d635e0 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 2018-09-19 09:30:25 -07:00
Wei-Chiu Chuang
28ceb34a72 HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 2018-09-19 03:12:20 -07:00
Shweta
27978bcb66 HDFS-13833. Improve BlockPlacementPolicyDefault's consider load logic. Contributed by Shweta. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-18 20:23:50 -07:00
Mingliang Liu
6ff509c32a HDFS-11719. Arrays.fill() wrong index in BlockSender.readChecksum() exception handling. Contributed by Tao Zhang 2018-09-18 10:25:28 -07:00
Inigo Goiri
d1541932db HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 2018-09-17 14:42:03 -07:00
Anu Engineer
fdf5a3fd63 HDFS-13919. Documentation: Improper formatting in Disk Balancer for Settings. 
Contributed by Ayush Saxena.
 2018-09-17 10:08:23 -07:00
Wei-Chiu Chuang
a1de8cbac5 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-09-14 05:22:56 +08:00
Inigo Goiri
e1b242a984 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 2018-09-13 09:16:58 -07:00
Daniel Templeton
78bd3b1db9 HDFS-13846. Safe blocks counter is not decremented correctly if the block is striped 
(Contributed by Kitti Nanasi via Daniel Templeton)

Change-Id: Id41747a67dc946fdf0dbde90643bb1ea7e9e0f70
 2018-09-12 11:18:55 -07:00
Brahma Reddy Battula
b2432d254c HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 2018-09-12 06:30:57 +05:30
Xiao Chen
8e54da1511 HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 2018-09-10 22:34:23 -07:00
Yiqun Lin
0da49642fc HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu. 2018-09-10 13:57:36 +08:00
Vinayakumar B
eef3bafae8 HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena. 2018-09-10 09:22:59 +05:30
Vinayakumar B
30eceec342 HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 2018-09-10 09:10:51 +05:30
Hrishikesh Gadre
335a8139f5 HDFS-13820. Add an ability to disable CacheReplicationMonitor. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-07 14:59:06 -07:00
Sean Mackrory
410dd3faa5 HDFS-13744. OIV tool should better handle control characters present in file or directory names. Contributed by Zsolt Venczel. 2018-09-07 12:39:33 -06:00
Giovanni Matteo Fumarola
eca1a4bfe9 HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering. 2018-09-06 14:48:00 -07:00
Giovanni Matteo Fumarola
9af96d4ed4 HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak. 2018-09-05 10:50:25 -07:00
Hrishikesh Gadre
6ccb809c2d HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-04 21:53:42 -07:00
Vinayakumar B
3801436e49 HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena. 2018-09-03 14:37:57 +05:30
Wei-Chiu Chuang
8aa6c4f079 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 26c2a97c56.
 2018-08-30 11:44:20 -07:00
Vinayakumar B
c36d69a7b3 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 
(cherry picked from commit 65977e5d8124be2bc208af25beed934933f170b3)
 2018-08-30 22:15:51 +05:30
Yiqun Lin
582cb10ec7 HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 2018-08-30 11:21:13 +08:00
Zsolt Venczel
3e18b957eb HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 2018-08-28 15:13:43 -07:00
Shweta
33f42efc94 HDFS-13837. Enable debug log for LeaseRenewer in TestDistributedFileSystem. Contributed by Shweta. 2018-08-28 13:56:32 -07:00
Wei-Chiu Chuang
26c2a97c56 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-08-27 16:02:35 -07:00
Yiqun Lin
b9b964d253 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-27 14:55:46 +08:00
Surendra Singh Lilhore
96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Vinayakumar B
770d9d9bb6 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:33:19 +05:30
Giovanni Matteo Fumarola
79c97f6a0b HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He. 2018-08-17 14:40:00 -07:00
Márton Elek
77b015000a HADOOP-8807. Update README and website to reflect HADOOP-8662. Contributed by Andras Bokor. 2018-08-17 11:10:29 +02:00
Xiao Chen
c67b0650ea HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 2018-08-16 23:14:21 -07:00
Wei-Chiu Chuang
1290e3c647 HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-16 16:29:38 -07:00
Daniel Templeton
8512e1a91b HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
 2018-08-16 15:00:45 -07:00
Yiqun Lin
6df606f1b4 HDFS-13829. Remove redundant condition judgement in DirectoryScanner#scan. Contributed by liaoyuxiangqin. 2018-08-16 18:44:18 +08:00
Xiao Chen
7dc79a8b5b HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-15 13:53:47 -07:00
Xiao Chen
ef02f9b233 HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin. 2018-08-15 09:23:05 -07:00
Akira Ajisaka
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Daniel Templeton
4a5006b1d0 HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky 
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
 2018-08-14 17:22:15 -07:00
Wei-Chiu Chuang
363bd16e31 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 17:19:00 -07:00
Xiao Chen
cede33997f HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 2018-08-14 11:57:22 -07:00
Wei-Chiu Chuang
61a9b4f58b HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 2018-08-14 11:51:27 -07:00
Wei-Chiu Chuang
4023eeba05 HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 2018-08-13 17:36:13 -07:00
Wei-Chiu Chuang
23854443ef HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 2018-08-13 16:12:37 -07:00
Arpit Agarwal
f760a544a7 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:35:19 -07:00
drankye
475bff6e8e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-13 17:32:56 +08:00
Uma Maheswara Rao Gangumalla
3ac07b720b HDFS-13808: [SPS]: Remove unwanted FSNamesystem #isFileOpenedForWrite() and #getFileInfo() function. Contributed by Rakesh R. 2018-08-12 03:06:07 -07:00
Uma Maheswara Rao G
39ed3a66db HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
dfcb331ba3 HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
db3f227d8a HDFS-13076: [SPS]: Resolve conflicts after rebasing HDFS-10285 branch to trunk. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Uma Maheswara Rao G
66e8f9b315 HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
2acc50b826 HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Surendra Singh Lilhore
75ccc1396b HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Rakesh Radhakrishnan
8467ec24fb HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier. Contributed by Rakesh R. 2018-08-12 03:06:05 -07:00
Uma Maheswara Rao G
4402f3f855 HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh. 2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
d3de4fb2a0 HDFS-13077. [SPS]: Fix review comments of external storage policy satisfier. Contributed by Rakesh R. 2018-08-12 03:06:04 -07:00
Rakesh Radhakrishnan
5845c36c16 HDFS-13050: [SPS]: Create start/stop script to start external SPS process. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
99594b48b8 HDFS-13075. [SPS]: Provide External Context implementation. Contributed by Uma Maheswara Rao G. 2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
3b83110d5e HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R. 2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
b0cb8d9bb4 HDFS-13033: [SPS]: Implement a mechanism to do file block movements for external SPS. Contributed by Rakesh R. 2018-08-12 03:06:03 -07:00
Rakesh Radhakrishnan
3159b39cf8 HDFS-13025. [SPS]: Implement a mechanism to scan the files for external SPS. Contributed by Uma Maheswara Rao G. 2018-08-12 03:06:03 -07:00
Rakesh Radhakrishnan
8d4f74e733 HDFS-12911. [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations. Contributed by Uma Maheswara Rao G 2018-08-12 03:06:03 -07:00
Surendra Singh Lilhore
05d4daf6ba HDFS-12982 : [SPS]: Reduce the locking and cleanup the Namesystem access. Contributed by Rakesh R. 2018-08-12 03:06:02 -07:00
Uma Maheswara Rao G
78420719eb HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R. 2018-08-12 03:06:02 -07:00
Surendra Singh Lilhore
c561cb316e HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
9b83f94f35 HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits. Contributed by Rakesh R. 2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
68017e3349 HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:02 -07:00
Surendra Singh Lilhore
5780f0624d HDFS-12556: [SPS] : Block movement analysis should be done in read lock. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
00eceed233 HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
bfd3f8bd8a HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
7ea24fc06c HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
0e820f16af HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier feature. Contributed by Rakesh R. 2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
3b601f2c0e HDFS-12152: [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time. Contributed by Rakesh R. 2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
4bcf61c696 HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R. 2018-08-12 03:06:00 -07:00
Rakesh Radhakrishnan
9e82e5a86e HDFS-12146. [SPS]: Fix TestStoragePolicySatisfierWithStripedFile#testSPSWhenFileHasLowRedundancyBlocks. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:00 -07:00
Rakesh Radhakrishnan
68af4e199a HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G 2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
5eb24ef7e7 HDFS-11264: [SPS]: Double checks to ensure that SPS/Mover are not running together. Contributed by Rakesh R. 2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
0b360b16ab HDFS-11965: [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
00cf207192 HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. Contributed by Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
46f7523eb8 HDFS-11966. [SPS] Correct the log in BlockStorageMovementAttemptedItems#blockStorageMovementResultCheck. Contributed by Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
20f9c62336 HDFS-11726. [SPS]: StoragePolicySatisfier should not select same storage type as source and destination in same datanode. Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
d638a7dc03 HDFS-11762. [SPS]: Empty files should be ignored in StoragePolicySatisfier. Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Uma Maheswara Rao G
e53f89ccc3 HDFS-11883: [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called. Contributed by Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Uma Maheswara Rao G
5ce332dc9a HDFS-11695: [SPS]: Namenode failed to start while loading SPS xAttrs from the edits log. Contributed by Surendra Singh Lilhore. 2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
6fe6c549e8 HDFS-11572. [SPS]: SPS should clean Xattrs when no blocks required to satisfy for a file. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
695a402fca HDFS-11334: [SPS]: NN switch and rescheduling movements can lead to have more than one coordinator for same file blocks. Contributed by Rakesh R. 2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
11a08a7e8f HDFS-11338: [SPS]: Fix timeout issue in unit tests caused by longger NN down time. Contributed by Wei Zhou and Rakesh R 2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
c00be44463 HDFS-11336: [SPS]: Remove xAttrs when movements done or SPS disabled. Contributed by Yuanbo Liu. 2018-08-12 03:05:58 -07:00
Rakesh Radhakrishnan
9b15f5418d HDFS-11239: [SPS]: Check Mover file ID lease also to determine whether Mover is running. Contributed by Wei Zhou 2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
e34331c31d HDFS-11243. [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
f8fc96a66e HDFS-11309. [SPS]: chooseTargetTypeInSameNode should pass accurate block size to chooseStorage4Block while choosing target. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
681d2804c9 HDFS-11186. [SPS]: Daemon thread of SPS should start only in Active NN. Contributed by Wei Zhou 2018-08-12 03:05:57 -07:00
Uma Maheswara Rao G
6215e35bb6 HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu 2018-08-12 03:05:57 -07:00
Uma Maheswara Rao G
df2b551e79 HDFS-11293: [SPS]: Local DN should be given preference as source node, when target available in same node. Contributed by Yuanbo Liu and Uma Maheswara Rao G 2018-08-12 03:05:56 -07:00
Rakesh Radhakrishnan
422f870607 HDFS-11289. [SPS]: Make SPS movement monitor timeouts configurable. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:56 -07:00
Uma Maheswara Rao G
81ed364b63 HDFS-11193 : [SPS]: Erasure coded files should be considered for satisfying storage policy. Contributed by Rakesh R 2018-08-12 03:05:56 -07:00
Uma Maheswara Rao G
b7bed9f00a HDFS-11248: [SPS]: Handle partial block location movements. Contributed by Rakesh R 2018-08-12 03:05:55 -07:00
Uma Maheswara Rao G
d81611fe55 HDFS-11032: [SPS]: Handling of block movement failure at the coordinator datanode. Contributed by Rakesh R 2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
5179d99b7e HDFS-11123. [SPS] Make storage policy satisfier daemon work on/off dynamically. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
cd5262aba0 HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou 2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
b07291e176 HDFS-11151. [SPS]: StoragePolicySatisfier should gracefully handle when there is no target node with the required storage type. Contributed by Rakesh R 2018-08-12 03:05:54 -07:00
Rakesh Radhakrishnan
b67ae6d9d7 HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu 2018-08-12 03:05:54 -07:00
Uma Maheswara Rao G
19b5aee3e4 HDFS-11068: [SPS]: Provide unique trackID to track the block movement sends to coordinator. Contributed by Rakesh R 2018-08-12 03:05:54 -07:00
Rakesh Radhakrishnan
047526b4c2 HDFS-11029. [SPS]:Provide retry mechanism for the blocks which were failed while moving its storage at DNs. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:53 -07:00
Rakesh Radhakrishnan
0f2d1ddc2c HDFS-10954. [SPS]: Provide mechanism to send blocks movement result back to NN from coordinator DN. Contributed by Rakesh R 2018-08-12 03:05:53 -07:00
Uma Maheswara Rao G
24add8c2f8 HDFS-10884: [SPS]: Add block movement tracker to track the completion of block movement future tasks at DN. Contributed by Rakesh R 2018-08-12 03:05:53 -07:00
Rakesh Radhakrishnan
e2a15d18bb HDFS-10801. [SPS]: Protocol buffer changes for sending storage movement commands from NN to DN. Contributed by Rakesh R 2018-08-12 03:05:53 -07:00
Uma Maheswara Rao G
1438da4944 HDFS-10800: [SPS]: Daemon thread in Namenode to find blocks placed in other storage than what the policy specifies. Contributed by Uma Maheswara Rao G 2018-08-12 03:05:53 -07:00
Kai Zheng
5692887395 HDFS-10794. [SPS]: Provide storage policy satisfy worker at DN for co-ordinating the block storage movement work. Contributed by Rakesh R 2018-08-12 03:05:52 -07:00
Chao Sun
a2a8c48699 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 2018-08-10 15:59:39 -07:00
Virajith Jalaparti
15241c6349 HDFS-13795. Fix potential NPE in InMemoryLevelDBAliasMapServer. 2018-08-10 09:38:40 -07:00
Chen Liang
5326a7906d HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-09 10:11:47 -07:00
Wangda Tan
49c687608b Make 3.1.1 awared by other branches 2018-08-08 13:02:12 -07:00
Xiao Chen
9499df7b81 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-08 10:40:20 -07:00
Márton Elek
36c0d742d4 HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr. 2018-08-08 17:31:55 +02:00
Ewan Higgs
2ec97abb2e HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran. 2018-08-08 13:50:23 +02:00
Vinayakumar B
4203bc738c HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena 2018-08-08 12:42:20 +05:30
Xiao Chen
6677717c68 HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell. 2018-08-07 22:05:17 -07:00
Vinayakumar B
2b0f977241 HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena. 2018-08-08 07:53:17 +05:30
Xiao Chen
0f8cb127cd HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization 
between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
 2018-08-07 16:13:41 -07:00
Virajith Jalaparti
b1a59b1644 HDFS-13796. Allow verbosity of InMemoryLevelDBAliasMapServer to be configurable. 2018-08-07 10:15:28 -07:00
Takanobu Asanuma
2b18bb4f37 HDFS-11610. sun.net.spi.nameservice.NameService has moved to a new location. Contributed by Akira Ajisaka. 2018-08-03 23:38:05 +09:00