big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
15,628 Commits 2 Branches 2 Tags 264 MiB
52b00600df
Commit Graph

6108 Commits

Author SHA1 Message Date
Andrew Wang
132f758e3d HDFS-11426. Refactor EC CLI to be similar to storage policies CLI. 2017-02-23 16:00:11 -08:00
Jing Zhao
13d4bcfe35 HDFS-4025. QJM: Sychronize past log segments to JNs that missed them. Contributed by Hanisha Koneru. 2017-02-22 16:33:38 -08:00
Andrew Wang
d150f061f4 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu. 2017-02-22 15:16:09 -08:00
Ming Ma
cfcd527323 HDFS-11411. Avoid OutOfMemoryError in TestMaintenanceState test runs. (Manoj Govindassamy via mingma) 2017-02-22 09:41:07 -08:00
Mingliang Liu
003ae00693 HDFS-11430. Separate class InnerNode from class NetworkTopology and make it extendable. Contributed by Tsz Wo Nicholas Sze 2017-02-21 15:32:46 -08:00
Andrew Wang
8ef7ebbc71 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu. 2017-02-21 14:54:20 -08:00
Andrew Wang
fc9ad3ce3a HDFS-11405. Rename "erasurecode" CLI subcommand to "ec". Contributed by Manoj Govindassamy. 2017-02-21 13:55:27 -08:00
Eric Payne
a07ddef101 HDFS-11404. Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc. Contributed by Eric Badger 2017-02-21 12:04:25 -06:00
Vinayakumar B
4804050630 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore) 2017-02-21 18:13:19 +05:30
Xiao Chen
02c549484a HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp. 2017-02-16 18:07:55 -08:00
Wei-Chiu Chuang
5690b51ef7 HDFS-11100. Recursively deleting file protected by sticky bit should fail. Contributed by John Zhuge. 2017-02-16 05:39:37 -08:00
Ming Ma
a136936d01 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma) 2017-02-15 20:24:07 -08:00
Mingliang Liu
0741dd3b9a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore 2017-02-15 11:37:26 -08:00
Jing Zhao
627da6f717 HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao. 2017-02-15 10:44:37 -08:00
Wei-Chiu Chuang
859bd159ae HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang. 2017-02-15 02:50:35 -08:00
Akira Ajisaka
8acb376c9c HDFS-11238. Fix checkstyle warnings in NameNode#createNameNode. Contributed by Ethan Li. 2017-02-15 16:53:50 +09:00
Yongjun Zhang
8e53f2b9b0 HDFS-11391. Numeric usernames do no work with WebHDFS FS write access. (Pierre Villard via Yongjun Zhang) 2017-02-14 13:40:53 -08:00
Wei-Chiu Chuang
0cf5993712 HDFS-11084. Add a regression test for sticky bit support of OIV ReverseXML processor. Contributed by Wei-Chiu Chuang. 2017-02-14 09:11:55 -08:00
Yiqun Lin
4164a2032a HDFS-11408. The config name of balance bandwidth is out of date. Contributed by Yiqun Lin. 2017-02-14 18:57:20 +08:00
Yiqun Lin
719df99c05 HDFS-11407. Document the missing usages of OfflineImageViewer processors. Contributed by Yiqun Lin. 2017-02-14 18:48:08 +08:00
Chris Douglas
4ed33e9ca3 HDFS-11026. Convert BlockTokenIdentifier to use Protobuf. Contributed by Ewan Higgs. 2017-02-13 11:29:18 -08:00
Kihwal Lee
07a5184f74 HDFS-11379. DFSInputStream may infinite loop requesting block locations. Contributed by Daryn Sharp. 2017-02-10 12:27:08 -06:00
Xiao Chen
2007e0cf2a HDFS-11210. Enhance key rolling to guarantee new KeyVersion is returned from generateEncryptedKeys after a key is rolled. 2017-02-07 20:36:11 -08:00
Xiao Chen
69b23632c4 HDFS-10860. Switch HttpFS from Tomcat to Jetty. Contributed by John Zhuge. 2017-02-07 09:36:55 -08:00
Yiqun Lin
663e683adf HDFS-10219. Change the default value for dfs.namenode.reconstruction.pending.timeout-sec from -1 to 300. Contributed by Yiqun Lin. 2017-02-06 19:19:30 +08:00
Yiqun Lin
9cbbd1eae8 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 2017-02-06 13:15:16 +08:00
Yiqun Lin
3433f572fa HDFS-11353. Improve the unit tests relevant to DataNode volume failure testing. Contributed by Yiqun Lin. 2017-02-02 19:38:17 +08:00
Arpit Agarwal
6aa09dc28a HDFS-11112. Journal Nodes should refuse to format non-empty directories. Contributed by Yiqun Lin. 2017-02-01 16:51:58 -08:00
Jing Zhao
b6f290d5b6 HDFS-11370. Optimize NamenodeFsck#getReplicaInfo. Contributed Takanobu Asanuma. 2017-02-01 11:21:35 -08:00
Lei Xu
bec9b7aa1d HDFS-11335. Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf. (Manoj Govindassamy via lei) 2017-02-01 14:42:51 +08:00
Ming Ma
312b36d113 HDFS-11378. Verify multiple DataNodes can be decommissioned/maintenance at the same time. (Manoj Govindassamy via mingma) 2017-01-27 16:16:42 -08:00
Arpit Agarwal
7bc333ad41 HDFS-11374. Skip FSync in CreateEditsLog to speed up edit log generation. Contributed by Hanisha Koneru. 2017-01-26 16:21:25 -08:00
Masatake Iwasaki
55c9f6d747 HDFS-9884. Use doxia macro to generate in-page TOC of HDFS site documentation. (iwasakims) 2017-01-27 08:06:39 +09:00
Xiao Chen
88da9f6b67 Revert "HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang." 
This reverts commit ada233b7cd.
 2017-01-26 13:42:50 -08:00
Xiao Chen
44606aa850 HDFS-11364. Add a test to verify Audit log entries for setfacl/getfacl commands over FS shell. Contributed by Manoj Govindassamy. 2017-01-26 10:48:26 -08:00
Arpit Agarwal
7c1cc30b3c HDFS-11369. Change exception message in StorageLocationChecker. 2017-01-26 09:12:38 -08:00
Andrew Wang
ff02bdfe65 Add CHANGES, RELEASENOTES, and jdiff for 3.0.0-alpha2 release. 2017-01-25 15:39:40 -08:00
Jing Zhao
b782bf2156 HDFS-11124. Report blockIds of internal blocks for EC files in Fsck. Contributed by Takanobu Asanuma. 2017-01-25 11:16:17 -08:00
Zhe Zhang
18e1d68209 HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki. 2017-01-25 09:58:39 -08:00
Arpit Agarwal
b57368b6f8 HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 16:58:20 -08:00
Wei-Chiu Chuang
355b907c09 HDFS-11121. Add assertions to BlockInfo#addStorage to protect from breaking reportedBlock-blockGroup mapping. Contributed by Takanobu Asanuma. 2017-01-21 05:56:22 +08:00
Ming Ma
f3fb94be05 HDFS-11296. Maintenance state expiry should be an epoch time and not jvm monotonic. (Manoj Govindassamy via mingma) 2017-01-19 22:31:15 -08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Lei Xu
78ae2aed8f HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-19 16:28:31 +08:00
Andrew Wang
a8f1c7f542 HDFS-110268. Correctly reconstruct erasure coding file from FSImage. Contributed by SammiChen. 2017-01-19 00:24:11 -08:00
Lei Xu
63320d1daa HDFS-11287. Storage class member storageDirs should be private to avoid unprotected access by derived classes. (Manoj Govindassamy via lei) 2017-01-19 15:38:20 +08:00
Yiqun Lin
724bb86351 HDFS-11316. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails in trunk. Contributed by Yiqun Lin. 2017-01-19 12:05:13 +08:00
Allen Wittenauer
0eb4b513b7 HADOOP-13673. Update scripts to be smarter when running with privilege 
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
 2017-01-18 14:39:05 -08:00
Andrew Wang
a2a5d7b5bc HDFS-10759. Change fsimage bool isStriped from boolean to an enum. Contributed by Ewan Higgs. 2017-01-18 13:31:05 -08:00
Vinitha Reddy Gankidi
8a0fa0f7e8 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:46:32 -08:00
Akira Ajisaka
9130af3f74 HADOOP-13955. Replace deprecated HttpServer2 and SSLFactory constants. Contributed by John Zhuge. 2017-01-18 18:11:33 +09:00
Akira Ajisaka
b1a9ec856b HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen. 2017-01-18 17:54:33 +09:00
Xiaoyu Yao
b1fce2b8b1 HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. Contributed by Xiaoyu Yao. 2017-01-17 14:33:26 -08:00
Arpit Agarwal
89bb05d92b HDFS-11339. Fix import. 2017-01-16 23:05:28 -08:00
Akira Ajisaka
e407449ddb HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 2017-01-17 10:10:23 +09:00
Arpit Agarwal
79e939d0b8 HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. Contributed by Hanisha Koneru. 2017-01-16 14:43:29 -08:00
Arpit Agarwal
a853b4e1b5 HDFS-11342. Fix FileInputStream leak in loadLastPartialChunkChecksum. Contributed by Chen Liang. 2017-01-16 13:45:22 -08:00
Lei Xu
2f8e9b7e4b Revert "HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei)" 
This reverts commit c18590fce2.
 2017-01-16 15:11:53 +08:00
Anu Engineer
2604e82eba HDFS-11344. The default value of the setting dfs.disk.balancer.block.tolerance.percent is different. Contributed by Yiqun Lin. 2017-01-13 22:38:46 -08:00
Wei-Chiu Chuang
1cde954a4f HDFS-11306. Print remaining edit logs from buffer if edit log can't be rolled. Contributed by Wei-Chiu Chuang. 2017-01-13 11:46:30 -08:00
Andrew Wang
5f336512d0 HADOOP-13961. Fix compilation failure from missing hadoop-kms test jar. Contributed by Sangjin Lee and John Zhuge. 2017-01-11 15:48:50 -08:00
Brahma Reddy Battula
a5ec1e31a3 HDFS-9935. Remove LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD and unused import from HdfsServerConstants. Contributed Yiqun Lin. 2017-01-11 19:34:01 +05:30
Ming Ma
467f5f1735 HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj Govindassamy via mingma) 2017-01-10 20:12:42 -08:00
Andrew Wang
e69231658d HDFS-11072. Add ability to unset and change directory EC policy. Contributed by Sammi Chen. 2017-01-10 11:32:48 -08:00
Arpit Agarwal
4046794a53 HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-10 10:43:11 -08:00
Lei Xu
c18590fce2 HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-10 19:37:58 +08:00
Jing Zhao
7ec609b289 HDFS-11273. Move TransferFsImage#doGetUrl function to a Util class. Contributed by Hanisha Koneru. 2017-01-09 18:05:33 -08:00
Yongjun Zhang
603cbcd513 HDFS-11292. log lastWrittenTxId etc info in logSyncAll. Contributed by Yongjun Zhang. 2017-01-09 15:44:42 -08:00
Arpit Agarwal
91bf504440 HDFS-11301. Double wrapping over RandomAccessFile in LocalReplicaInPipeline#createStreams. Contributed by Hanisha Koneru. 2017-01-09 10:32:18 -08:00
Chris Nauroth
4c8f9e1302 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 2017-01-05 15:04:47 -08:00
Xiaoyu Yao
87bb1c49bb HDFS-11279. Cleanup unused DataNode#checkDiskErrorAsync(). Contributed by Hanisha Koneru 2017-01-03 18:25:46 -08:00
Andrew Wang
7fcc73fc0d HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 2017-01-03 09:58:00 -08:00
Xiao Chen
e769957556 HDFS-11275. Check groupEntryIndex and throw a helpful exception on failures when removing ACL. 2016-12-29 10:36:50 -08:00
Lei Xu
a4f66655ec HDFS-11267. Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei) 2016-12-29 16:57:40 +08:00
Lei Xu
e9f1396834 HDFS-11251. ConcurrentModificationException during DataNode#refreshVolumes. (Manoj Govindassamy via lei) 2016-12-29 15:11:25 +08:00
Xiaoyu Yao
603f3ef138 HDFS-11274. Datanode should only check the failed volume upon IO errors. Contributed by Xiaoyu Yao. 2016-12-28 22:08:13 -08:00
Daniel Templeton
ce3613c965 HDFS-11278. Add missing @Test annotation for TestSafeMode.testSafeModeUtils() (Contributed by Lukas Majercak via Daniel Templeton) 2016-12-28 20:09:00 -08:00
Brahma Reddy Battula
0ddb8defad HDFS-11252. TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException. Contributed by Yiqun Lin. 2016-12-28 10:34:33 +05:30
Daniel Templeton
e216e8e233 HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-27 13:03:16 -08:00
Mingliang Liu
c0e0ef2969 HDFS-11270. Document the missing options of NameNode bootstrap command. Contributed by Yiqun Lin 2016-12-27 11:21:10 -08:00
Akira Ajisaka
ea547529cb HDFS-11272. Refine the assert messages in TestFSDirAttrOp. Contributed by Jimmy Xiang. 2016-12-26 17:15:45 +09:00
Akira Ajisaka
8f218ea284 HDFS-11250. Fix a typo in ReplicaUnderRecovery#setRecoveryID. Contributed by Yiqun Lin. 2016-12-26 16:51:29 +09:00
Akira Ajisaka
483cd06ad4 HDFS-11271. Typo in NameNode UI. Contributed by Wei-Chiu Chuang. 2016-12-26 16:36:54 +09:00
Xiaoyu Yao
4e9029653d HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2016-12-22 23:46:58 -08:00
Kai Zheng
56a13a6a59 HDFS-11216. Add remoteBytesRead counter metrics for erasure coding reconstruction task. Contributed by Sammi Chen 2016-12-22 14:18:54 +08:00
Haohui Mai
f6e80acd68 HDFS-11258. File mtime change could not save to editlog. Contributed by Jimmy Xiang. 2016-12-21 13:04:03 -08:00
Xiaoyu Yao
f678080dbd HDFS-11182. Update DataNode to use DatasetVolumeChecker. Contributed by Arpit Agarwal. 2016-12-20 13:53:32 -08:00
Xiaoyu Yao
5daa8d8631 HDFS-10913. Introduce fault injectors to simulate slow mirrors. Contributed by Xiaobing Zhou. 2016-12-20 13:17:52 -08:00
Xiao Chen
4af66b1d66 HDFS-11247. Add a test to verify NameNodeMXBean#getDecomNodes() and Live/Dead Decom Nodes shown in NameNode WebUI. Contributed by Manoj Govindassamy. 2016-12-20 12:42:13 -08:00
Xiao Chen
5b7acdd206 HDFS-11195. Return error when appending files by webhdfs rest api fails. Contributed by Yuanbo Liu. 2016-12-20 12:24:00 -08:00
Brahma Reddy Battula
1b401f6a73 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage compression. Contributed by Brahma Reddy Battula. 2016-12-20 20:54:03 +05:30
Xiaoyu Yao
fe4ff64a4a HDFS-10959. Adding per disk IO statistics and metrics in DataNode. Contributed by Xiaoyu Yao. 2016-12-19 22:42:54 -08:00
Akira Ajisaka
d2026ae9e6 HDFS-11262. Remove unused variables in FSImage.java. Contributed by Jagadesh Kiran N. 2016-12-20 14:34:35 +09:00
Anu Engineer
a95639068c HDFS-9911. TestDataNodeLifeline Fails intermittently. Contributed by Yiqun Lin 2016-12-16 09:46:21 -08:00
Xiao Chen
cee0c468b0 HDFS-11188. Change min supported DN and NN versions back to 2.x. Contributed by Andrew Wang. 2016-12-15 16:48:50 -08:00
Wei-Chiu Chuang
aebb9127ba HDFS-11160. VolumeScanner reports write-in-progress replicas as corrupt incorrectly. Contributed by Wei-Chiu Chuang and Yongjun Zhang. 2016-12-15 16:32:50 -08:00
Mingliang Liu
6fce191549 HDFS-11253. FileInputStream leak on failure path in BlockSender. Contributed by Arpit Agarwal 2016-12-15 15:39:15 -08:00
Mingliang Liu
ac0d402c6a HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-15 14:53:53 -08:00
Akira Ajisaka
36947f79ba HDFS-11249. Redundant toString() in DFSConfigKeys.java. Contributed by Jagadesh Kiran N. 2016-12-16 00:33:27 +09:00
Arpit Agarwal
6ba9587d37 HDFS-10958. Add instrumentation hooks around Datanode disk IO. 2016-12-14 11:18:58 -08:00