big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,534 Commits 2 Branches 2 Tags 264 MiB
84f7638840
Commit Graph

8069 Commits

Author SHA1 Message Date
Inigo Goiri
24080666e5 HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun. 2019-12-16 18:41:45 -08:00
Masatake Iwasaki
dc6cf17b34 HDFS-15048. Fix findbug in DirectoryScanner. 2019-12-16 19:00:57 +09:00
Ayush Saxena
7a87007545 HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina. 2019-12-15 00:33:32 +05:30
Chen Liang
65c4660bcd HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 2019-12-12 10:22:05 -08:00
Masatake Iwasaki
bdd00f10b4
HDFS-15047. Document the new decommission monitor (HDFS-14854). (#1755) 2019-12-13 00:35:41 +09:00
Takanobu Asanuma
93bb368094 HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li 2019-12-12 10:56:19 +09:00
Erik Krogen
c174d50b30 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 2019-12-11 09:39:37 -08:00
Stephen O'Donnell
c93cb6790e HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-10 17:16:43 -08:00
Masatake Iwasaki
11cd5b6e39 HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-12-07 08:46:45 +09:00
Wei-Chiu Chuang
ecd461f940 HDFS-14751. Synchronize on diffs in DirectoryScanner. Contributed by Lisheng Sun. 2019-12-06 15:10:40 -08:00
Wei-Chiu Chuang
313b76f8e9 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-12-06 15:10:40 -08:00
Ayush Saxena
705b172b95 HDFS-14998. [SBN read] Update Observer Namenode doc for ZKFC after HDFS-14130. Contributed by Fei Hui. 2019-12-06 23:04:09 +05:30
Ayush Saxena
83a14559e5 HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui. 2019-12-05 18:52:01 +05:30
Ayush Saxena
0c217feed8 HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun. 2019-12-03 09:18:09 +05:30
Ayush Saxena
6b2d6d4aaf HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina. 2019-11-29 23:55:37 +05:30
Surendra Singh Lilhore
44f7b9159d HDFS-15013. Reduce NameNode overview tab response time. Contributed by HuangTao. 2019-11-29 00:42:31 +05:30
Surendra Singh Lilhore
0384687811 HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-28 23:49:35 +05:30
Ayush Saxena
46166bd8d1 HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena. 2019-11-28 18:28:20 +05:30
Yiqun Lin
c3659f8f94 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. 2019-11-28 14:41:49 +08:00
Yiqun Lin
2b452b4e60 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 2019-11-28 10:43:35 +08:00
Yiqun Lin
c8bef4d6a6 HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. 2019-11-27 10:57:20 +08:00
Ayush Saxena
049940e77b HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:30:02 +05:30
Takanobu Asanuma
b25e94ce29 HDFS-14924. RenameSnapshot not updating new modification time. Contributed by hemanthboyina 2019-11-22 13:48:02 +09:00
Yiqun Lin
9b6906fe91 HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun. 2019-11-22 10:53:55 +08:00
Surendra Singh Lilhore
26270196a2 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:31:25 +05:30
Ayush Saxena
98d249dcda HDFS-14996. RBF: GetFileStatus fails for directory with EC policy set in case of multiple destinations. Contributed by Ayush Saxena. 2019-11-21 12:08:42 +05:30
Surendra Singh Lilhore
fd264b8265 HDFS-14995. Use log variable directly instead of passing as argument in InvalidateBlocks#printBlockDeletionTime(). Contributed by Lisheng Sun. 2019-11-21 00:00:42 +05:30
He Xiaoqiao
0b50aa29fd HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
 2019-11-19 12:16:57 -08:00
Surendra Singh Lilhore
c870514740 HDFS-14992. TestOfflineEditsViewer is failing in Trunk. Contributed by hemanthboyina. 2019-11-20 00:07:43 +05:30
Ayush Saxena
34cb595d2e HDFS-14967. TestWebHDFS fails in Windows. Contributed by Renukaprasad C. 2019-11-18 18:49:58 +05:30
Yiqun Lin
b3119b9ab6 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. 2019-11-16 11:32:41 +08:00
Hui Fei
67f2c491fe HDFS-14802. The feature of protect directories should be used in RenameOp (#1669) 2019-11-15 12:35:37 -08:00
He Xiaoqiao
c892a879dd HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
 2019-11-15 12:16:31 -08:00
Erik Krogen
b2cc8b6b4a HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 2019-11-15 10:10:37 -08:00
Erik Krogen
586defe711 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 2019-11-13 11:46:35 -08:00
Chao Sun
97ec34e117 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 2019-11-12 11:35:50 -08:00
Inigo Goiri
40150da1e1 HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina. 2019-11-12 10:53:54 -08:00
Ayush Saxena
77934bc07b HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui. 2019-11-11 11:05:34 +05:30
Takanobu Asanuma
6663d6a5c2 HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li 2019-11-11 14:04:57 +09:00
Surendra Singh Lilhore
320008bb7c HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 2019-11-09 22:14:35 +05:30
Ayush Saxena
247584eb63 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:49:01 +05:30
Chen Liang
dd900259c4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 09:56:19 -08:00
Chen Liang
9d0d580031 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 09:30:04 -08:00
Ayush Saxena
9e287054a8 HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 2019-11-06 22:02:03 +05:30
Ayush Saxena
dcf55838ae HDFS-14880. Correct the sequence of statistics & exit message in balencer. Contributed by Renukaprasad C. 2019-11-06 21:45:03 +05:30
Surendra Singh Lilhore
c36014165c HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 2019-11-06 19:28:55 +05:30
Inigo Goiri
bfb8f28cc9 HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang. 2019-11-05 09:19:52 -08:00
Ayush Saxena
b643a1cbe8 HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun. 2019-11-05 09:23:20 +05:30
Ayush Saxena
2ffec347eb HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-05 01:37:15 +05:30
Wei-Chiu Chuang
eb73ba6ed5 HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 2019-11-04 09:40:42 -08:00
Zhao Junwang
de6b8b0c0b HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
 2019-11-01 16:37:53 -07:00
Ayush Saxena
7d7acb004a HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:50:10 +05:30
Surendra Singh Lilhore
02009c3bb7 HDFS-14768. EC : Busy DN replica should be consider in live replica check. Contributed by guojh. 2019-11-01 22:15:05 +05:30
Ayush Saxena
9d25ae7669 HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:49:37 +05:30
Ayush Saxena
d9fbedc4ae HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun. 2019-10-31 23:39:27 +05:30
Jonathan Hung
e6137d0e50 Make upstream aware of 2.10.0 release 2019-10-30 20:58:32 -07:00
Ayush Saxena
fa4904cdca HDFS-14935. Refactor DFSNetworkTopology#isNodeInScope. Contributed by Lisheng Sun. 2019-10-29 12:30:24 +05:30
Eric Yang
30ed24a421 HDFS-14730. Removed unused configuration dfs.web.authentication.filter. 
Contributed by Chen Zhang
 2019-10-28 19:27:32 -04:00
Eric Badger
9ef6ed9c1c HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 2019-10-28 16:37:04 +00:00
zhaoyim
30db895b59 HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-25 13:09:14 -07:00
Takanobu Asanuma
862526530a HDFS-14933. Fixing a typo in documentation of Observer NameNode. Contributed by Xieming Li 2019-10-25 15:23:05 +09:00
Takanobu Asanuma
0db0f1e399 HDFS-14917. Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html". Contributed by Xieming Li 2019-10-25 10:27:38 +09:00
Wei-Chiu Chuang
a1b4eebcc9
HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2019-10-24 13:09:52 -07:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Vinayakumar B
ee699dc26c HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:14:09 +05:30
Mukul Kumar Singh
a901405ad8 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-22 13:31:15 -07:00
Ayush Saxena
6020505943 HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:44:22 +05:30
Ayush Saxena
19f35cfd57 HDFS-14918. Remove useless getRedundancyThread from BlockManagerTestUtil. Contributed by Fei Hui. 2019-10-22 18:34:02 +05:30
Wei-Chiu Chuang
72003b19bf HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-21 17:31:43 -07:00
Kihwal Lee
c4e27ef773 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 2019-10-21 16:33:15 -05:00
Ayush Saxena
74c2329fc3 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:35:12 +05:30
Ayush Saxena
447f46d962 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:10:59 +05:30
Yuming Wang
3d41f33018 HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng. 
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-18 08:25:02 -07:00
Surendra Singh Lilhore
54dc6b7d72 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 2019-10-17 22:28:47 +05:30
Ayush Saxena
5527d79adb HDFS-14810. Review FSNameSystem editlog sync. Contributed by Xiaoqiao He. 2019-10-17 21:56:30 +05:30
Surendra Singh Lilhore
336abbd873 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 2019-10-14 23:31:43 +05:30
Ashvin
fabd41fa48 HDFS-14856. Fetch file ACLs while mounting external store. (#1478) 2019-10-14 09:44:56 -07:00
Ayush Saxena
5f4641a120 HDFS-14238. A log in NNThroughputBenchmark should change log level to INFO instead of ERROR. Contributed by Shen Yinjie. 2019-10-12 21:06:23 +05:30
Surendra Singh Lilhore
d76e2655ac HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 2019-10-09 23:20:28 +05:30
Chen Liang
72ae371e7a HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-08 11:56:52 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Surendra Singh Lilhore
382967be51 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-08 00:14:30 +05:30
Eric Yang
aa24add8f0 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle
 2019-10-04 13:13:10 -04:00
Takanobu Asanuma
b23bdaf085 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 2019-10-04 16:47:06 +09:00
Stephen O'Donnell
c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00
Virajith Jalaparti
844b766da5 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)" 2019-10-03 21:32:48 -07:00
Ayush Saxena
a3fe404244 HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C. 2019-10-03 22:34:55 +05:30
Chen Liang
1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Akira Ajisaka
3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li. 2019-10-02 11:03:16 +09:00
Wei-Chiu Chuang
6ef6594c7e HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
 2019-10-01 21:16:46 +05:30
Konstantin V Shvachko
b3275ab1f2 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:48:10 -07:00
Konstantin V Shvachko
760b523e58 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-29 13:11:31 -07:00
Ayush Saxena
2200871607 HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun. 2019-09-28 22:03:53 +05:30
Ayush Saxena
c4c8d5fd0e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-28 21:44:44 +05:30
Ayush Saxena
0d5d0b914a Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao." 
This reverts commit ce58c05f1d.
 2019-09-28 21:32:15 +05:30
Sahil Takiar
13b427fc05 HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable (#963) Contributed by Sahil Takiar. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
 2019-09-27 12:25:03 -07:00
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Zhankun Tang
2870668cfa Make upstream aware of 3.1.3 release. 2019-09-27 16:44:07 +08:00
Ayush Saxena
a3f68933d7 HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui. 2019-09-26 19:54:07 +05:30
Ayush Saxena
66400c1cbb HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 2019-09-25 01:31:15 +05:30