big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,139 Commits 2 Branches 2 Tags 264 MiB
2eba262472
Commit Graph

8530 Commits

Author SHA1 Message Date
Ayush Saxena
66400c1cbb HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 2019-09-25 01:31:15 +05:30
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Inigo Goiri
afa1006a53 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor. 2019-09-24 09:58:42 -07:00
Inigo Goiri
816d3cb908 HDFS-14837. Review of Block.java. Contributed by David Mollitor. 2019-09-24 09:54:09 -07:00
Wei-Chiu Chuang
43203b466d HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-24 09:38:58 -07:00
Ayush Saxena
c30e495557 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-23 21:22:50 +05:30
Vinayakumar B
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494). Contributed by Vinayakumar B. 2019-09-23 12:37:47 +05:30
Rohith Sharma K S
4c0a7a9e13 Make upstream aware of 3.2.1 release. 2019-09-23 06:20:54 +00:00
Rakesh Radhakrishnan
659c88801d HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 2019-09-22 22:02:54 +05:30
Ayush Saxena
aa938662f9 HDFS-14833. RBF: Router Update Doesn't Sync Quota. Contributed by Ayush Saxena. 2019-09-21 09:53:13 +05:30
Inigo Goiri
3f223bebfa HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun. 2019-09-20 11:45:01 -07:00
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Inigo Goiri
a79f286c6f HDFS-14609. RBF: Security should use common AuthenticationFilter. Contributed by Chen Zhang. 2019-09-19 11:00:10 -07:00
Malcolm Taylor
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM. 
Contributed by Malcolm Taylor.

Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
 2019-09-19 16:00:05 +01:00
Chen Liang
4ed0aefe9f HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 16:47:46 -07:00
Wei-Chiu Chuang
5dd859a8a0 HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-09-18 09:34:01 -07:00
Sahil Takiar
3cf6e4272f HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs 
Signed-off-by: Anu Engineer <aengineer@apache.org>
 2019-09-17 15:51:39 -07:00
Inigo Goiri
f580a87079 HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun. 2019-09-17 14:55:34 -07:00
Ayush Saxena
e04b8a46c3 HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina. 2019-09-14 10:39:20 +05:30
Stephen O'Donnell
06ad6540b3 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-13 10:36:45 -07:00
Surendra Singh Lilhore
4852a90e4b HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 2019-09-13 00:17:13 +05:30
Surendra Singh Lilhore
2ff2a7f612 HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina. 2019-09-12 19:43:18 +05:30
Surendra Singh Lilhore
d1c303a497 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 2019-09-12 19:11:50 +05:30
Akira Ajisaka
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor. 2019-09-12 12:41:57 +09:00
Takanobu Asanuma
c255333e20 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 2019-09-11 16:54:08 +09:00
Takanobu Asanuma
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414) 2019-09-11 10:32:07 +09:00
Erik Krogen
c92a3e94d8 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2019-09-06 10:22:28 -07:00
Surendra Singh Lilhore
d98c54816d HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green. 2019-09-06 18:12:42 +05:30
Surendra Singh Lilhore
fa7f03fc56 HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C. 2019-09-06 17:55:30 +05:30
Inigo Goiri
494d75eb2b HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang. 2019-09-05 21:15:17 -07:00
Inigo Goiri
72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Ayush Saxena
2f70b52a5b HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq. 2019-09-05 08:29:56 +05:30
Surendra Singh Lilhore
05704754a0 HDFS-14777. RBF: Set ReadOnly is failing for mount Table but actually readonly succeed to set. Contributed by Ranith Sardar. 2019-09-04 15:09:09 +05:30
Xiaoyu Yao
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 2019-09-03 09:29:58 -07:00
Ayush Saxena
91b01a1db7 HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena. 2019-09-03 18:08:42 +05:30
Surendra Singh Lilhore
5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
Stephen O'Donnell
915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-02 09:47:04 -07:00
Wei-Chiu Chuang
d207aba026 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit 7bebad61d9.
 2019-09-02 09:43:44 -07:00
Ayush Saxena
040f6e93bb HDFS-14654. RBF: TestRouterRpc#testNamenodeMetrics is flaky. Contributed by Chen Zhang. 2019-09-02 13:01:52 +05:30
Ayush Saxena
c3abfcefdd HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena. 2019-09-02 07:24:04 +05:30
Ayush Saxena
18d74fe41c HDFS-14711. RBF: RBFMetrics throws NullPointerException if stateStore disabled. Contributed by Chen Zhang. 2019-09-01 12:53:10 +05:30
Surendra Singh Lilhore
96f7dc1992 HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui. 2019-08-30 14:31:05 +05:30
Ayush Saxena
e220dac15c HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:40:13 +05:30
leosunli
915271245b Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-08-29 18:35:26 -07:00
Stephen O'Donnell
7bebad61d9 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 17:38:07 -07:00
Istvan Fajth
fcb7884bfc HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 16:56:14 -07:00
Inigo Goiri
3b22fcd377 HDFS-14256. Review Logging of NameNode Class. Contributed by David Mollitor. 2019-08-29 14:21:09 -07:00
Inigo Goiri
ffca734c62 HDFS-14104. Review getImageTxIdToRetain. Contributed by David Mollitor. 2019-08-29 14:18:35 -07:00
He Xiaoqiao
f600fbb6c4 HDFS-11246. FSNameSystem#logAuditEvent should be called outside the read or write locks. Contributed by He Xiaoqiao, Kuhu Shukla. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Kuhu Shukla <kshukla@apache.org>
 2019-08-29 10:10:52 -07:00
Ayush Saxena
8e779a151e HDFS-14721. RBF: ProxyOpComplete is not accurate in FederationRPCPerformanceMonitor. Contributed by xuzq. 2019-08-29 20:08:38 +05:30
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
Inigo Goiri
48cb583906 HDFS-14710. RBF: Improve some RPC performance by using previous block. Contributed by xuzq. 2019-08-28 10:48:00 -07:00
CR Hota
6e37d65b03 HDFS-14760. Log INFO mode if snapshot usage and actual usage differ. Contributed by CR Hota. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 16:20:25 -07:00
He Xiaoqiao
dde9399b37 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 15:26:59 -07:00
Shashikant Banerjee
ce8eb1283a Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee." 
This reverts commit 62445021d5 as it has unintended changes in DirectoryWithSnapshotFeature class..
 2019-08-27 23:23:44 +05:30
Jonathan Hung
8ab7020e64 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 2019-08-27 10:30:31 -07:00
Takanobu Asanuma
b69ac575a1 HDFS-14772. RBF: hdfs-rbf-site.xml can't be loaded automatically. Contributed by Yuxuan Wang 2019-08-27 14:32:32 +09:00
Arpit Agarwal
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 2019-08-26 15:57:55 -07:00
Ayush Saxena
d2225c8ca8 HDFS-14722. RBF: GetMountPointStatus should return mountTable information when getFileInfoAll throw IOException. Contributed by xuzq. 2019-08-24 08:47:43 +05:30
Inigo Goiri
0b796754b9 HDFS-14766. RBF: MountTableStoreImpl#getMountTableEntries returns extra entry. Contributed by Chen Zhang. 2019-08-23 14:53:57 -07:00
Chen Liang
ebef99dcf4 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:18:43 -07:00
Erik Krogen
d699022fce HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 2019-08-23 11:33:21 -07:00
Wang Yuxuan
894e2300d6 HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly (#1334) 
HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly
 2019-08-23 10:46:39 -07:00
Ayush Saxena
c92de8209d HDFS-14747. RBF: IsFileClosed should be return false when the file is open in multiple destination. Contributed by xuzq. 2019-08-23 19:08:07 +05:30
Akira Ajisaka
bd7baea5a5
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 2019-08-23 15:28:27 +09:00
Akira Ajisaka
abc8fde4ca
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 2019-08-23 14:44:34 +09:00
Stephen O'Donnell
b67812ea21 HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index (#1028). Contributed by Stephen O'Donnell. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-08-22 17:09:57 -07:00
Stephen O'Donnell
93daf69f90 HDFS-14675. Increase Balancer Defaults Further. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Gabor Bota <gabota@apache.org>
 2019-08-22 15:21:33 -07:00
Inigo Goiri
e04dcfdc57 HDFS-14583. FileStatus#toString() will throw IllegalArgumentException. Contributed by xuzq. 2019-08-22 10:22:38 -07:00
Surendra Singh Lilhore
76790a1e67 HDFS-14358. Provide LiveNode and DeadNode filter in DataNode UI. Contributed by hemanthboyina. 2019-08-22 12:27:45 +05:30
Ayush Saxena
52c77bc160 HDFS-14741. RBF: RecoverLease should be return false when the file is open in multiple destination. Contributed by xuzq 2019-08-22 08:57:22 +05:30
Ayush Saxena
f9029c4070 HDFS-14744. RBF: Non secured routers should not log in error mode when UGI is default. Contributed by CR Hota. 2019-08-22 02:40:39 +05:30
Ayush Saxena
0f598aed13 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 02:25:06 +05:30
Inigo Goiri
93595febaa HDFS-14756. RBF: getQuotaUsage may ignore some folders. Contributed by Chen Zhang. 2019-08-21 09:39:57 -07:00
Inigo Goiri
5eeb6da2d4 HDFS-14714. RBF: implement getReplicatedBlockStats interface. Contributed by Chen Zhang. 2019-08-21 09:38:17 -07:00
Wei-Chiu Chuang
57f7370174 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit b58a35f374.
 2019-08-21 08:59:49 -07:00
Anu Engineer
8aaf5e1a14 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.
 2019-08-20 20:24:19 -07:00
Surendra Singh Lilhore
3a145e2918 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-20 15:55:26 -07:00
Sandeep Kumar
269b543367 HADOOP-16523. Minor spell mistake in comment : Line number 101 (#388) 2019-08-20 13:36:23 -07:00
Wei-Chiu Chuang
4cb22cd867 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:37:11 -07:00
Wei-Chiu Chuang
aa6995fde2 HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 2019-08-20 10:32:55 -07:00
Sunil G
bd92462321 HDFS-14729. Upgrade Bootstrap and jQuery versions used in HDFS UIs. Contributed by Vivek Ratnavel Subramanian. This closes #1297 2019-08-20 18:42:19 +05:30
Shashikant Banerjee
62445021d5 HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee. 2019-08-20 14:37:01 +05:30
Wei-Chiu Chuang
360a96f342 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 2019-08-19 13:08:55 -07:00
HUAN-PING SU
abae6ff2a2 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-19 11:14:51 -07:00
Surendra Singh Lilhore
b8db5b9a98 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 18:13:35 -07:00
Wei-Chiu Chuang
b58a35f374 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:46:00 -07:00
Wei-Chiu Chuang
d873ddd656 HDFS-14127. Add a description about the observer read configuration. Contributed by xiangheng. 2019-08-17 04:39:22 -07:00
Wei-Chiu Chuang
528378784f HDFS-12012. Fix spelling mistakes in BPServiceActor.java. Contributed by chencan. 2019-08-17 04:37:05 -07:00
hunshenshi
a38b9e137e HDFS-14456:HAState#prepareToEnterState neednt a lock (#770) Contributed by hunshenshi. 2019-08-16 14:53:06 -07:00
LeonGao91
9a1d8cfaf5 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 2019-08-16 08:00:51 -07:00
Surendra Singh Lilhore
34681643e9 HDFS-14719. Correct the safemode threshold value in BlockManagerSafeMode. Contributed by hemanthboyina. 2019-08-15 12:12:18 +05:30
Shashikant Banerjee
0a85af959c HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 2019-08-15 10:16:25 +05:30
Ayush Saxena
b06c2345ef HDFS-14713. RBF: RouterAdmin supports refreshRouterArgs command but not on display. Contributed by wangzhaohui. 2019-08-14 20:44:31 +05:30
Siyao Meng
3c0382f1b9 HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2019-08-14 07:30:35 -07:00
Wei-Chiu Chuang
c13ec7ab66 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 2019-08-13 17:15:26 -07:00
Wei-Chiu Chuang
633b7c1cfe HDFS-14625. Make DefaultAuditLogger class in FSnamesystem to Abstract. Contributed by hemanthboyina. 2019-08-13 16:50:49 -07:00
Masatake Iwasaki
da0006fe04 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-14 08:39:40 +09:00
Siyao Meng
6ae8bc3a4a HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng. 2019-08-13 16:27:57 -07:00
Ayush Saxena
e9b6b81de4 HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng. 2019-08-13 19:17:10 +05:30