big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
15,659 Commits 2 Branches 2 Tags 264 MiB
15e3873dc3
Commit Graph

4313 Commits

Author SHA1 Message Date
Xiao Chen
cd5e10ccca HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge. 2016-08-29 17:48:08 -07:00
Yongjun Zhang
5d1609ddf2 HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin. 2016-08-29 13:59:54 -07:00
Yongjun Zhang
c25817159a HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang. 2016-08-27 22:51:31 -07:00
Zhe Zhang
e806db7190 HDFS-10584. Allow long-running Mover tool to login with keytab. Contributed by Rakesh R. 2016-08-26 16:43:45 -07:00
Andrew Wang
a445b82baa HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy. 2016-08-26 15:39:18 -07:00
Zhe Zhang
407b519fb1 HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 2016-08-26 14:19:55 -07:00
Kihwal Lee
8b7adf4ddf HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp. 2016-08-26 15:39:56 -05:00
Kihwal Lee
a1f3293762 HDFS-10772. Reduce byte/string conversions for get listing. Contributed by Daryn Sharp. 2016-08-24 15:21:08 -05:00
Daryn Sharp
ec252ce0fc HDFS-10762. Pass IIP for file status related methods 2016-08-24 08:46:47 -05:00
Wei-Chiu Chuang
f0efea490e HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-23 04:14:24 -07:00
Akira Ajisaka
e90f3359de HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 2016-08-23 19:57:23 +09:00
Eric Yang
c49333becf HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) 2016-08-22 18:29:56 -07:00
Kihwal Lee
3ca4d6ddfd Revert "HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp." 
This reverts commit 22fc46d765.
 2016-08-22 16:57:45 -05:00
Kihwal Lee
22fc46d765 HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp. 2016-08-22 15:37:02 -05:00
Jing Zhao
0faee62a0c HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 2016-08-19 22:13:36 -07:00
Kihwal Lee
2550371f66 HDFS-10711. Optimize FSPermissionChecker group membership check. Contributed by Daryn Sharp. 2016-08-19 09:12:17 -05:00
Akira Ajisaka
8179f9a493 HDFS-10645. Make block report size as a metric and add this metric to datanode web ui. Contributed by Yuanbo Liu. 2016-08-19 16:15:00 +09:00
Arpit Agarwal
ca13e7971d HDFS-10773. BlockSender should not synchronize on the dataset object. (Contributed by Chen Liang) 2016-08-17 16:29:08 -07:00
Kihwal Lee
869393643d HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 2016-08-17 15:53:03 -05:00
Anu Engineer
b047bc7270 HDFS-10560. DiskBalancer: Reuse ObjectMapper instance to improve the performance. Contributed by Yiqun Lin. 2016-08-16 10:20:08 -07:00
Mingliang Liu
ef55fe1716 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:23:47 -07:00
Anu Engineer
5628b36c08 HDFS-10559. DiskBalancer: Use SHA1 for Plan ID. Contributed by Xiaobing Zhou. 2016-08-15 20:10:21 -07:00
Anu Engineer
02abd131b8 HDFS-10567. Improve plan command help message. Contributed by Xiaobing Zhou. 2016-08-15 19:58:57 -07:00
Kihwal Lee
864f878d59 HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 2016-08-15 17:28:09 -05:00
Kihwal Lee
03dea65e0b HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 2016-08-15 16:45:44 -05:00
Anu Engineer
bed69d18e6 HDFS-10580. DiskBalancer: Make use of unused methods in GreedyPlanner to print debug info. Contributed by Yiqun Lin 2016-08-15 12:40:29 -07:00
Kihwal Lee
83e57e083f HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 2016-08-15 13:01:23 -05:00
Anu Engineer
9f29f423e4 HDFS-10737. disk balancer add volume path to report command. Contributed by Yuanbo Liu. 2016-08-15 09:47:30 -07:00
Wei-Chiu Chuang
9019606b69 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 2016-08-12 05:58:13 -07:00
Kai Zheng
b5af9be72c HDFS-8668. Erasure Coding: revisit buffer used for encoding and decoding. Contributed by Sammi Chen 2016-08-13 13:52:37 +08:00
Wei-Chiu Chuang
a428d4f50e HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 2016-08-10 22:17:35 -07:00
Xiaoyu Yao
ec289bbece HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:49:36 -07:00
Xiaoyu Yao
fdafd0cb4f Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca7.
 2016-08-10 15:47:55 -07:00
Xiaoyu Yao
9f816d7ca7 Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:14:38 -07:00
Yuanbo Liu
10e84c6a6e HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 2016-08-10 11:07:42 -07:00
Wei-Chiu Chuang
d00d3add9e HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-09 18:11:34 -07:00
Lei Xu
9c6a4383ca HDFS-10681. DiskBalancer: query command should report Plan file path apart from PlanID. (Manoj Govindassamy via lei) 2016-08-09 15:59:22 -07:00
Lei Xu
cc48251bfd HDFS-10457. DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) 2016-08-09 15:51:19 -07:00
Kihwal Lee
b10c936020 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 2016-08-09 11:56:21 -05:00
Arpit Agarwal
8c0638471f HADOOP-10682. Replace FsDatasetImpl object lock with a separate lock object. (Chen Liang) 2016-08-08 12:02:53 -07:00
Konstantin V Shvachko
4d3af47f27 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 2016-08-07 14:29:26 -07:00
Akira Ajisaka
ef432579a7 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 2016-08-05 17:31:11 +09:00
Wei-Chiu Chuang
18d9e6ec0b HDFS-10588. False alarm in datanode log - ERROR - Disk Balancer is not enabled. Contributed by Weiwei Yang. 2016-08-04 19:40:33 -07:00
Kihwal Lee
438a9f047e HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 2016-08-04 16:14:13 -05:00
Kihwal Lee
57369a678c HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 2016-08-04 14:55:21 -05:00
Tsz-Wo Nicholas Sze
8c9b44eaff In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:45:40 -07:00
Kihwal Lee
6ae39199da HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 2016-08-04 09:07:34 -05:00
Kihwal Lee
6f63566694 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 2016-08-03 15:20:20 -05:00
Jing Zhao
f4ba5ff1d7 HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 2016-08-03 12:42:05 -07:00
Kihwal Lee
22ef5286bc HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 2016-08-03 13:12:43 -05:00
Kihwal Lee
58db263e93 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 2016-08-03 12:14:59 -05:00
Kihwal Lee
bebf10d245 HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 2016-08-03 11:53:41 -05:00
Akira Ajisaka
a5fb298e56 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-02 17:07:59 +09:00
Konstantin V Shvachko
c4463f2ef2 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 85a20508bd.
 2016-08-01 22:34:40 -07:00
Daryn Sharp
9f473cf903 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 2016-08-01 10:14:28 -05:00
Yongjun Zhang
8d32bd884d HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 2016-07-29 23:20:14 -07:00
Akira Ajisaka
204a2055b1 HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 2016-07-29 15:39:48 +09:00
Xiaoyu Yao
ce3d68e9c3 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:07:45 -07:00
Lei Xu
67406460f0 HDFS-4176. EditLogTailer should call rollEdits with a timeout. (Lei Xu) 2016-07-28 13:32:00 -07:00
Akira Ajisaka
414fbfab41 HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 2016-07-28 16:58:23 +09:00
Andrew Wang
098ec2b11f HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou. 2016-07-27 17:55:41 -07:00
Yongjun Zhang
eb7ff0c992 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 2016-07-27 13:58:04 -07:00
Wei-Chiu Chuang
d84ab8a578 HDFS-10598. DiskBalancer does not execute multi-steps plan. Contributed by Lei (Eddy) Xu. 2016-07-26 16:01:50 -07:00
Wei-Chiu Chuang
25f0a9b3ab HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 2016-07-26 08:01:42 -07:00
Vinitha Reddy Gankidi
85a20508bd HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:50:59 -07:00
Jing Zhao
0cde9e12a7 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 2016-07-25 18:41:13 -07:00
Kai Zheng
b79ba4fe96 HDFS-10651. Clean up some configuration related codes about legacy block reader. Contributed by Youwei Wang 2016-07-23 17:45:33 +08:00
Arpit Agarwal
c6e3a0020f HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-22 12:33:50 -07:00
Xiao Chen
132deb4cac HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 2016-07-21 16:41:02 -07:00
Jing Zhao
bd3dcf46e2 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 2016-07-21 11:14:39 -07:00
Akira Ajisaka
38128baff4 HDFS-10425. Clean up NNStorage and TestSaveNamespace. Contributed by Andras Bokor. 2016-07-20 10:44:11 -07:00
Akira Ajisaka
f7dabe3add HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 2016-07-19 15:13:01 -07:00
Akira Ajisaka
c48e9d608c HDFS-10628. Log HDFS Balancer exit message to its own log. Contributed by Jiayi Zhou. 2016-07-15 11:42:44 -07:00
Lei Xu
382dff7475 HDFS-10600. PlanCommand#getThrsholdPercentage should not use throughput value. (Yiqun Lin via lei) 2016-07-14 10:40:34 -07:00
Kihwal Lee
2bbc3ea1b5 HDFS-10617. PendingReconstructionBlocks.size() should be synchronized. Contributed by Eric Badger. 2016-07-13 21:59:35 -05:00
Zhe Zhang
087290e6b1 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 2016-07-12 23:18:37 -07:00
Jitendra Pandey
c447efebdb HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 2016-07-11 14:55:33 -07:00
Yongjun Zhang
da6f1b88dd HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 2016-07-08 19:40:44 -07:00
Akira Ajisaka
8b4b5259d5 HDFS-10589. Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params. Contributed by shenyinjie. 2016-07-04 07:15:04 -07:00
Akira Ajisaka
2e835d986e HDFS-10572. Fix TestOfflineEditsViewer#testGenerated. Contributed by Surendra Singh Lilhore. 2016-07-03 21:11:56 -07:00
Kai Zheng
8b281bce85 HDFS-10548. Remove the long deprecated BlockReaderRemote. Contributed by Kai Zheng 2016-07-03 11:56:23 +08:00
Colin Patrick Mccabe
e4a2545620 HDFS-9805. Add server-side configuration for enabling TCP_NODELAY for DataTransferProtocol and default it to true (Gary Helmling via cmccabe) 2016-06-29 12:41:42 -07:00
Andrew Wang
c0829f4493 HDFS-6434. Default permission for creating file should be 644 for WebHdfs/HttpFS. Contributed by Wellington Chevreuil. 2016-06-28 13:55:26 -07:00
Allen Wittenauer
23c3ff85a9 HADOOP-13209. replace slaves with workers (John Smith via aw) 2016-06-28 05:53:03 -07:00
Vinayakumar B
2a0082c51d HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 2016-06-28 16:49:39 +05:30
Vinayakumar B
73615a789d HDFS-10536. Standby NN can not trigger log roll after EditLogTailer thread failed 3 times in EditLogTailer.triggerActiveLogRoll method (Contributed by XingFeng Shen 2016-06-26 15:45:32 +05:30
Sangjin Lee
bf74dbf80d HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 2016-06-24 14:44:15 -07:00
Zhe Zhang
22be604fa3 Revert "HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki." 
This reverts commit 0424056a77.
 2016-06-24 09:36:03 -07:00
Zhe Zhang
0424056a77 HDFS-10534. NameNode WebUI should display DataNode usage rate with a certain percentile. Contributed by Kai Sasaki. 2016-06-24 08:58:29 -07:00
Anu Engineer
8a6e354122 HDFS-10552. DiskBalancer "-query" results in NPE if no plan for the node. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Anu Engineer
e8de28181a HDFS-10551. o.a.h.h.s.diskbalancer.command.Command does not actually verify options as expected. Contributed by Anu Engineer. 2016-06-23 18:27:02 -07:00
Arpit Agarwal
66fa34c839 HDFS-10557. Fix handling of the -fs Generic option. (Arpit Agarwal) 2016-06-23 18:27:01 -07:00
Anu Engineer
7b23ad1ef7 HDFS-10550. DiskBalancer: fix issue of order dependency in iteration in ReportCommand test. Contributed by Xiaobing Zhou. 2016-06-23 18:27:01 -07:00
Anu Engineer
0774412e41 HDFS-10545. DiskBalancer: PlanCommand should use -fs instead of -uri to be consistent with other hdfs commands. Contributed by Anu Engineer. 2016-06-23 18:27:01 -07:00
Anu Engineer
5b8e1c26d7 HDFS-10541. Diskbalancer: When no actions in plan, error message says "Plan was generated more than 24 hours ago". Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer
cb68e5b3bd HDFS-10540. Diskbalancer: The CLI error message for disk balancer is not enabled is not clear. Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer
3225c24e0e HDFS-10520. DiskBalancer: Fix Checkstyle issues in test code. Contributed by Anu Engineer. 2016-06-23 18:27:00 -07:00
Anu Engineer
7e2be5c4a0 HDFS-10518. DiskBalancer: Pretty-print json in Query command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
af11ab34d0 HDFS-10517. DiskBalancer: Support help command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
b502102bb1 HDFS-9461. DiskBalancer: Add Report Command. Contributed by Xiaobing Zhou. 2016-06-23 18:21:08 -07:00
Anu Engineer
121142cf95 HDFS-10501. DiskBalancer: Use the default datanode port if port is not provided. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
78a1032b71 HDFS-10500. Diskbalancer: Print out information when a plan is not generated. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
d2ff793ae4 HDFS-10496. DiskBalancer: ExecuteCommand checks planFile in a wrong way. Contributed by Lei (Eddy) Xu. 2016-06-23 18:21:08 -07:00
Anu Engineer
64ccb232cc HDFS-10478. DiskBalancer: resolve volume path names. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
47dcb0f952 HDFS-10476. DiskBalancer: Plan command output directory should be a sub-directory. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
43eee50966 HDFS-10403. DiskBalancer: Add cancel command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Arpit Agarwal
9e5fcb5e40 HDFS-10402. DiskBalancer: Add QueryStatus command. (Contributed by Anu Engineer) 2016-06-23 18:21:08 -07:00
Anu Engineer
5df2d2b8fd HDFS-10399. DiskBalancer: Add JMX for DiskBalancer. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
1b39b283c7 HDFS-9546: DiskBalancer: Add Execute command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
75882ec0b0 HDFS-9545: DiskBalancer: Add Plan Command. Contributed by Anu Engineer. 2016-06-23 18:21:08 -07:00
Anu Engineer
1594b472bb HDFS-9543. DiskBalancer: Add Data mover. Contributed by Anu Engineer. 2016-06-23 18:20:24 -07:00
Anu Engineer
7820737cfa HDFS-9735. DiskBalancer : Refactor moveBlockAcrossStorage to be used by disk balancer. Contributed by Anu Engineer. 2016-06-23 18:20:24 -07:00
Anu Engineer
050677077b HDFS-9720. DiskBalancer : Add configuration parameters. Contributed by Anu Engineer. 2016-06-23 18:18:48 -07:00
Anu Engineer
6c606bf5c8 HDFS-9709. DiskBalancer : Add tests for disk balancer using a Mock Mover class. Contributed by Anu Engineer. 2016-06-23 18:18:48 -07:00
Anu Engineer
9be9703716 Fix a build break in HDFS-1312 2016-06-23 18:18:48 -07:00
Anu Engineer
ec601673a8 HDFS-10170. DiskBalancer: Force rebase diskbalancer branch. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Arpit Agarwal
75a711a2d5 HDFS-9703. DiskBalancer: getBandwidth implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
918722bdd2 HDFS-9702. DiskBalancer: getVolumeMap implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Anu Engineer
747227e9de HDFS-9817. Use SLF4J in new classes. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Arpit Agarwal
9847640603 HDFS-9683. DiskBalancer: Add cancelPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
e646c2eb50 HDFS-9681. DiskBalancer: Add QueryPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
2b1b2faf76 HDFS-9671. DiskBalancer: SubmitPlan implementation. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
66f0bb646d HDFS-9647. DiskBalancer: Add getRuntimeSettings. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
96fe685b7a HDFS-9645. DiskBalancer: Add Query RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
0501d430e2 HDFS-9595. DiskBalancer: Add cancelPlan RPC. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
7100c0da35 HDFS-9588. DiskBalancer: Add submitDiskbalancer RPC. (Anu Engineer via Arpit Agarwal) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
5724a10316 HDFS-9469. DiskBalancer: Add Planner. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Arpit Agarwal
e325c6ade9 HDFS-9611. DiskBalancer : Replace htrace json imports with jackson. (Contributed by Anu Engineer) 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
599eca07b4 HDFS-9526. Fix jackson annotation imports. Contributed by Xiaobing Zhou 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
30c6ebd699 HDFS-9449. DiskBalancer: Add connectors. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Tsz-Wo Nicholas Sze
91a5c48143 HDFS-9420. Add DataModels for DiskBalancer. Contributed by Anu Engineer 2016-06-23 18:18:48 -07:00
Xiaoyu Yao
dca298d79e HDFS-10469. Add number of active xceivers to datanode metrics. Contributed by Hanisha Koneru. 2016-06-23 14:13:39 -07:00
Kai Zheng
e6cb07520f HDFS-10460. Recompute block checksum for a particular range less than file size on the fly by reconstructing missed block. Contributed by Rakesh R 2016-06-24 17:39:32 +08:00
Uma Maheswara Rao G
01643020d2 HDFS-10555: Unable to loadFSEdits due to a failure in readCachePoolInfo. Contributed by Uma Maheswara Rao G 2016-06-22 22:06:13 -07:00
Uma Maheswara Rao G
17eae9ebb3 HDFS-10473: Allow only suitable storage policies to be set on striped files. Contributed by Uma Maheswara Rao G 2016-06-22 11:17:43 -07:00
Akira Ajisaka
d433b16ce6 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 2016-06-22 11:00:16 +09:00
Brahma Reddy Battula
f2ac132d6a HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 2016-06-21 15:42:28 +05:30
Colin Patrick Mccabe
46f1602e89 HDFS-10448. CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1 (Yiqun Lin via cmccabe) 2016-06-20 18:25:09 -07:00
Colin Patrick Mccabe
5f6b4157a4 HDFS-10328. Add per-cache-pool default replication num configuration (xupeng via cmccabe) 2016-06-20 10:42:50 -07:00
Arpit Agarwal
0319d73c3b HDFS-10437. ReconfigurationProtocol not covered by HDFSPolicyProvider. (Arpit Agarwal) 2016-06-18 16:16:54 -07:00
Kihwal Lee
280069510b HDFS-10493. Add links to datanode web UI in namenode datanodes page. Contributed by Weiwei Yang. 2016-06-17 11:05:44 -05:00
Colin Patrick Mccabe
2ca73445f5 HDFS-10525. Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap (Xiao Chen via cmccabe) 2016-06-15 22:47:01 -07:00
Colin Patrick Mccabe
2449db507d HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 2016-06-15 22:35:19 -07:00
Ming Ma
b48f27e794 HDFS-9922. Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes. (Chris Trezzo via mingma) 2016-06-15 22:00:52 -07:00
Ming Ma
7d521a29ee HDFS-9016. Display upgrade domain information in fsck. (mingma) 2016-06-14 20:05:50 -07:00
Andrew Wang
709a814fe0 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 2016-06-13 11:50:19 -07:00
Allen Wittenauer
d44f4745b4 HDFS-7987. Allow files / directories to be moved (Ravi Prakash via aw) 2016-06-10 09:02:28 -07:00
Ravi Prakash
ae047655f4 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 2016-06-08 13:44:22 -07:00
Akira Ajisaka
e620530301 HDFS-10485. Fix findbugs warning in FSEditLog.java. (aajisaka) 2016-06-07 17:52:55 +09:00
Zhe Zhang
6de9213df1 HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 2016-06-06 15:52:39 -07:00
Allen Wittenauer
99a771cd7a HDFS-7767. Use the noredirect flag in WebHDFS to allow web browsers to upload files via the NN UI (Ravi Prakash via aw) 2016-06-03 17:07:39 -07:00
Allen Wittenauer
15f018434c HDFS-9877. HDFS Namenode UI: Fix browsing directories that need to be encoded (Ravi Prakash via aw) 2016-06-03 17:06:29 -07:00
Arpit Agarwal
97e2449477 HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (Contributed by Akira Ajisaka) 2016-06-02 13:14:45 -07:00
Akira Ajisaka
1df6f5735c HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 2016-06-03 04:10:32 +09:00
Kai Zheng
d749cf65e1 HDFS-9833. Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data. Contributed by Rakesh R. 2016-06-02 12:56:21 +08:00
Colin Patrick Mccabe
c7921c9bdd HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 2016-05-31 17:37:52 -07:00
Kai Zheng
93d8a7f2a2 Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng" 
This reverts commit a6c79f92d5.
 2016-06-01 08:41:15 +08:00
Yongjun Zhang
5ea6fd85c7 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 2016-05-27 10:02:02 -07:00
Zhe Zhang
8c84a2a93c HDFS-10236. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-3]. Contributed by Rakesh R. 2016-05-26 16:50:55 -07:00
Kai Zheng
77202fa103 HADOOP-13010. Refactor raw erasure coders. Contributed by Kai Zheng 2016-05-27 13:23:34 +08:00
Kai Zheng
a6c79f92d5 HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng 2016-05-27 04:29:42 +08:00
Vinayakumar B
2dab64da2f HDFS-10217. show 'blockScheduled' tooltip in datanodes table. 2016-05-25 12:34:52 +05:30
Tsz-Wo Nicholas Sze
15ed080e36 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 12:49:48 -07:00
Allen Wittenauer
4b0f55b6ea HDFS-7766. Add a flag to WebHDFS op=CREATE to not respond with a 307 redirect (Ravi Prakash via aw) 2016-05-23 15:52:01 -07:00
Chris Nauroth
500e946729 HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 2016-05-20 13:01:43 -07:00
Chris Nauroth
bcde1562d2 HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 2016-05-20 12:47:41 -07:00
Kihwal Lee
204de430bf HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-19 15:52:32 -05:00
Andrew Wang
010e6ac328 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor. 2016-05-18 14:13:41 -07:00
Lei Xu
cf552aa87b HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei) 2016-05-18 09:38:37 -07:00
Junping Du
f6ef876fe1 MAPREDUCE-6657. Job history server can fail on startup when NameNode is in start phase. Contributed by Haibo Chen. 2016-05-17 14:41:47 -07:00
Yongjun Zhang
e24fe2641b HDFS-9732, Improve DelegationTokenIdentifier.toString() for better logging. Contributed by Yongjun Zhang. 2016-05-17 14:03:08 -07:00
Kihwal Lee
1356cbe994 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit 0942954e8a.
 2016-05-17 15:20:03 -05:00
Kihwal Lee
0942954e8a HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 2016-05-17 15:06:25 -05:00
Colin Patrick Mccabe
7cd5ae62f6 HDFS-10404. Fix formatting of CacheAdmin command usage help text (Yiqun Lin via cmccabe) 2016-05-17 11:10:48 -07:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Lei Xu
6a6e74acf5 HDFS-10410. RedundantEditLogInputStream.LOG is set to wrong class. (John Zhuge via lei) 2016-05-16 17:05:46 -07:00
Kai Zheng
ad9441122f HDFS-8449. Add tasks count metrics to datanode for ECWorker. Contributed by Bo Li. 2016-05-15 23:39:09 -07:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Zhe Zhang
307db61aac HDFS-10172. hdfs erasurecode command should remove the redundant -usage option. Contributed by Yuanbo Liu. 2016-05-10 15:20:17 -07:00
Colin Patrick Mccabe
f0252ad252 HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe) 2016-05-10 11:15:05 -07:00
Akira Ajisaka
411fb4b2b7 HDFS-10337. OfflineEditsViewer stats option should print 0 instead of null for the count of operations. Contributed by Yiqun Lin. 2016-05-08 15:49:13 -07:00
Xiaoyu Yao
dacd1f50fe HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 2016-05-05 18:08:47 -07:00
Ming Ma
1268cf5fbe HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 2016-05-04 17:02:26 -07:00
Arpit Agarwal
6d77d6eab7 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:52:43 -07:00
Ravi Prakash
2753185a01 HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano 2016-04-29 22:32:27 -07:00
Kihwal Lee
7da540d03e HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 2016-04-29 15:50:43 -05:00
Tsz-Wo Nicholas Sze
4da6f69ca1 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:22:33 -07:00
Kihwal Lee
6243eabb48 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla 2016-04-28 12:44:53 -05:00
Lei Xu
0a152103f1 HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 2016-04-27 14:22:51 -07:00
Kihwal Lee
919a1d824a HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 2016-04-27 08:19:48 -05:00
Andrew Wang
6be22ddbf1 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-26 13:04:18 -07:00
Kihwal Lee
c6a2430b47 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 2016-04-26 11:45:29 -05:00
Zhe Zhang
5865fe2bf0 HDFS-9869. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-2]. Contributed by Rakesh R. 2016-04-25 22:01:54 -07:00
Tsz-Wo Nicholas Sze
f308561f1d HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-04-25 12:01:48 -07:00
Jing Zhao
63e5412f1a HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou. 2016-04-22 15:14:40 -07:00
Ravi Prakash
d6402faded Revert "HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano" 
This reverts commit 4838b735f0.
 2016-04-22 10:17:20 -07:00
Kihwal Lee
5d4255a801 HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 2016-04-22 11:08:44 -05:00
Tsz-Wo Nicholas Sze
14ab7a81e2 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:41:50 -07:00
Tsz-Wo Nicholas Sze
bbce1d525e HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:20:12 -07:00
Ravi Prakash
4838b735f0 HADOOP-12563. Updated utility (dtutil) to create/modify token files. Contributed by Matthew Paduano 2016-04-21 11:32:39 -07:00
Xiaoyu Yao
b4be288c5d HDFS-10207. Support enable Hadoop IPC backoff without namenode restart. Contributed by Xiaobing Zhou. 2016-04-21 10:18:48 -07:00
Xiaoyu Yao
5bd7b592e5 HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao. 2016-04-20 23:10:52 -07:00
Chris Nauroth
63ac2db59a HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 2016-04-20 13:39:44 -07:00
Arpit Agarwal
af9bdbe447 HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:26:30 -07:00
Kai Zheng
b5d4c7dc76 HADOOP-12924. Configure raw erasure coders for supported codecs. Contributed by Rui Li. 2016-04-20 15:47:01 +08:00
Jing Zhao
be0bce1b71 HDFS-10306. SafeModeMonitor should not leave safe mode if name system is starting active service. Contributed by Mingliang Liu. 2016-04-18 18:05:49 -07:00
Colin Patrick Mccabe
cb3ca460ef HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 2016-04-18 13:47:56 -07:00
Kihwal Lee
d8b729e16f HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 2016-04-18 07:58:55 -05:00
Walter Su
67523ffcf4 HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 2016-04-18 09:28:02 +08:00
Kihwal Lee
c970f1d005 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang. 2016-04-14 12:45:47 -05:00
Kihwal Lee
0d1c1152f1 HDFS-10282. The VolumeScanner should warn about replica files which are misplaced. Contributed by Colin Patrick McCabe. 2016-04-14 07:58:24 -05:00
Xiaoyu Yao
5566177c9a HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode. Contributed by Xiaobing Zhou. 2016-04-13 16:51:13 -07:00
Xiaoyu Yao
4895c73dd4 Revert "HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode" 
This reverts commit 192112d5a2.
 2016-04-13 16:45:18 -07:00
Andrew Wang
314aa21a89 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 2016-04-13 16:39:50 -07:00
Xiaobing Zhou
192112d5a2 HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode 2016-04-13 15:34:41 -07:00
Colin Patrick Mccabe
35f0770555 HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe) 2016-04-12 16:28:42 -07:00
Zhe Zhang
6ef42873a0 HDFS-9918. Erasure Coding: Sort located striped blocks based on decommissioned states. Contributed by Rakesh R. 2016-04-12 13:38:58 -07:00
Colin Patrick Mccabe
600d129bb8 HDFS-10273. Remove duplicate logSync() and log message in FSN#enterSafemode() (Vinayakumar B via cmccabe) 2016-04-12 10:52:39 -07:00
Akira Ajisaka
209303be3a HDFS-8356. Document missing properties in hdfs-default.xml. Contributed by Ray Chiang. 2016-04-12 15:26:50 +09:00
Vinayakumar B
a9a607f8fc HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 2016-04-12 11:44:34 +05:30
Akira Ajisaka
9c32f8785e HDFS-10266. Remove unused properties dfs.client.file-block-storage-locations.num-threads and dfs.client.file-block-storage-locations.timeout.millis. Contributed by Ray Chiang. 2016-04-08 15:32:13 +09:00
Uma Maheswara Rao G
3c18a53cbd HDFS-9719. Refactoring ErasureCodingWorker into smaller reusable constructs. Contributed by Kai Zheng. 2016-04-06 22:50:24 -07:00
Tsz-Wo Nicholas Sze
654cd1d0c0 HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:38:42 +08:00
Colin Patrick Mccabe
4bd7cbc29d HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose 2016-04-06 21:07:31 -07:00
Eric Payne
aede8c10ec HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee 2016-04-06 20:20:14 +00:00
Colin Patrick Mccabe
188f65287d HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe) 2016-04-06 11:28:34 -07:00
Jing Zhao
221b3a8722 HDFS-10192. Namenode safemode not coming out during failover. Contributed by Brahma Reddy Battula. 2016-04-06 10:42:59 -07:00
Ravi Prakash
0cd320a846 HDFS-10235. Last contact for Live Nodes should be relative time. Contributed by Brahma Reddy Battula. 2016-04-05 13:41:19 -07:00
Vinayakumar B
818d6b799e HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula) 2016-04-05 09:49:39 +08:00
Colin Patrick Mccabe
f6b1a81812 HDFS-8496. Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe) 2016-04-04 18:02:15 -07:00
Kihwal Lee
a7d1fb0cd2 HDFS-10178. Permanent write failures can happen if pipeline recoveries occur for the first packet. Contributed by Kihwal Lee. 2016-04-04 16:40:00 -05:00
Allen Wittenauer
0a74610d1c HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw) 2016-03-31 07:51:05 -07:00
Colin Patrick Mccabe
37e23ce45c HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting connection timeouts (cmccabe) 2016-03-30 13:37:37 -07:00
Vinayakumar B
09d63d5a19 HDFS-5177. blocksScheduled count should be decremented for abandoned blocks (Contributed by Vinayakumar B) 2016-03-30 14:22:00 +08:00
Arpit Agarwal
ddfe6774c2 HDFS-9439. Support reconfiguring fs.protected.directories without NN restart. (Contributed by Xiaobing Zhou) 2016-03-29 13:55:00 -07:00
Vinayakumar B
1f004b3367 HDFS-9871. "Bytes Being Moved" -ve(-1 B) when cluster was already balanced. (Contributed by Brahma Reddy Battulla) 2016-03-29 11:25:41 +08:00
Uma Maheswara Rao G
3a4ff7776e HDFS-9694. Make existing DFSClient#getFileChecksum() work for striped blocks. Contributed by Kai Zheng 2016-03-26 19:58:09 -07:00
Arpit Agarwal
a337ceb74e Revert "HDFS-9694. Make existing DFSClient#getFileChecksum() work for striped blocks. Contributed by Kai Zheng" 
This reverts commit e5ff0ea7ba.
 2016-03-26 09:20:01 -07:00
Uma Maheswara Rao G
e5ff0ea7ba HDFS-9694. Make existing DFSClient#getFileChecksum() work for striped blocks. Contributed by Kai Zheng 2016-03-26 00:52:50 -07:00
Lei Xu
fde8ac5d85 Add missing files from HDFS-9005. (lei) 2016-03-25 17:11:35 -07:00
Lei Xu
4fcfea71bf HDFS-9005. Provide support for upgrade domain script. (Ming Ma via Lei Xu) 2016-03-25 17:09:12 -07:00
Allen Wittenauer
738155063e HADOOP-12857. rework hadoop-tools (aw) 2016-03-23 13:46:38 -07:00
Colin Patrick Mccabe
a7d8f2b396 HDFS-10189. PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE (Joe Pallas via cmccabe) 2016-03-22 14:56:02 -07:00
Colin Patrick Mccabe
680716f31e HDFS-9951. Use string constants for XML tags in OfflineImageReconstructor (Lin Yiqun via cmccabe) 2016-03-21 11:40:37 -07:00
Andrew Wang
e3bb38d625 HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed by Xiao Chen. 2016-03-21 11:39:05 -07:00
Haohui Mai
8a3f0cb255 HDFS-7166. SbNN Web UI shows #Under replicated blocks and #pending deletion blocks. Contributed by Wei-Chiu Chuang. 2016-03-20 17:21:25 -07:00
Sangjin Lee
cd8b6889a7 HDFS-9579. Provide bytes-read-by-network-distance metrics at FileSystem.Statistics level (Ming Ma via sjlee) 2016-03-19 14:02:04 -07:00
Akira Ajisaka
9b623fbaf7 HDFS-3677. dfs.namenode.edits.dir.required is missing from hdfs-default.xml. Contributed by Mark Yang. 2016-03-19 01:11:15 +09:00
Kihwal Lee
63c966a3fb HDFS-9874. Long living DataXceiver threads cause volume shutdown to block. Contributed by Rushabh Shah. 2016-03-18 10:24:59 -05:00
Andrew Wang
02a250db9f HDFS-10173. Typo in DataXceiverServer. Contributed by Michael Han. 2016-03-16 18:49:57 -07:00
Zhe Zhang
32d043d9c5 HDFS-9857. Erasure Coding: Rename replication-based names in BlockManager to more generic [part-1]. Contributed by Rakesh R. 2016-03-16 16:53:58 -07:00
Ravi Prakash
bd5556ba12 Download File from UI broken after pagination. Contributed by Brahma Reddy Battula 2016-03-14 14:30:13 -07:00
Chris Nauroth
5644137ada HDFS-9941. Do not log StandbyException on NN, other minor logging fixes. Contributed by Arpit Agarwal. 2016-03-14 09:54:54 -07:00
Arpit Agarwal
e01c6ea688 HDFS-1477. Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart. (Contributed by Xiaobing Zhou) 2016-03-10 19:03:55 -08:00
Colin Patrick Mccabe
79961ecea8 HDFS-9933. ReverseXML should be capitalized in oiv usage message (cmccabe) 2016-03-10 13:43:27 -08:00
Colin Patrick Mccabe
bd49354c6d HDFS-9934. ReverseXML oiv processor should bail out if the XML file's layoutVersion doesn't match oiv's (cmccabe) 2016-03-10 13:41:06 -08:00
Zhe Zhang
7600e3c48f HDFS-7866. Erasure coding: NameNode manages multiple erasure coding policies. Contributed by Rui Li. 2016-03-08 22:30:30 -08:00
Jing Zhao
743a99f2db HDFS-8786. Erasure coding: use simple replication for internal blocks on decommissioning datanodes. Contributed by Rakesh R. 2016-03-08 10:24:57 -08:00
Arpit Agarwal
c2140d05ef HDFS-9882. Add heartbeatsTotal in Datanode metrics. (Contributed by Hua Liu) 2016-03-07 21:10:24 -08:00
Arpit Agarwal
724d2299cd HDFS-9906. Remove spammy log spew when a datanode is restarted. (Contributed by Brahma Reddy Battula) 2016-03-07 12:19:53 -08:00
Harsh J
fd1c09be3e HDFS-9521. TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk. Contributed by Wellington Chevreuil 2016-03-07 17:14:51 +05:30
Chris Nauroth
2759689d7d HDFS-9239. DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness. Contributed by Chris Nauroth. 2016-03-04 15:29:50 -08:00
Arpit Agarwal
27941a1811 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-03-02 18:35:28 -08:00
Colin Patrick Mccabe
700b0e4019 HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from an XML file (cmccabe) 2016-03-02 17:56:11 -08:00
Xiaoyu Yao
5abf051249 HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed by Austin Donnelly and Chris Douglas. 2016-03-02 09:23:17 -08:00
Akira Ajisaka
27e0681f28 HDFS-9851. NameNode throws NPE when setPermission is called on a path that does not exist. Contributed by Brahma Reddy Battula. 2016-03-02 14:43:03 +09:00
Jing Zhao
f2ba7da4f0 HDFS-9876. shouldProcessOverReplicated should not count number of pending replicas. Contributed by Jing Zhao. 2016-03-01 18:41:57 -08:00
Kihwal Lee
2c8496ebf3 HDFS-8791. block ID-based DN storage layout can be very slow for datanode on ext4. Contributed by Chris Trezzo. 2016-03-01 15:04:59 -06:00
Chris Nauroth
2137e8feeb HDFS-9870. Remove unused imports from DFSUtil. Contributed by Brahma Reddy Battula. 2016-03-01 11:15:13 -08:00
Uma Maheswara Rao G
307ec80aca HDFS-9733. Refactor DFSClient#getFileChecksum and DataXceiver#blockChecksum. Contributed by Kai Zheng 2016-02-29 21:52:20 -08:00
Jing Zhao
2151716832 HDFS-7964. Add support for async edit logging. Contributed by Daryn Sharp. 2016-02-29 15:34:43 -08:00
Jing Zhao
408f2c807b HDFS-9866. BlockManager#chooseExcessReplicasStriped may weaken rack fault tolerance. Contributed by Jing Zhao. 2016-02-28 14:54:49 -08:00
Tsz-Wo Nicholas Sze
d1d4e16690 HDFS-9710. DN can be configured to send block receipt IBRs in batches. 2016-02-26 15:32:25 -08:00
Xiaoyu Yao
eab52dfb35 HDFS-9831. Document webhdfs retry configuration keys introduced by HDFS-5219/HDFS-5122. Contributed by Xiaobing Zhou. 2016-02-26 14:14:12 -08:00
Zhe Zhang
8808779db3 HDFS-9734. Refactoring of checksum failure report related codes. Contributed by Kai Zheng. 
Change-Id: Ie69a77e3498a360959f8e213c51fb2b17c28b64a
 2016-02-25 09:55:50 -08:00
Zhe Zhang
ccff6035f5 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
Change-Id: I54b82e0b11929082057b4d21bfcf285222c94cdb
 2016-02-25 09:24:32 -08:00
Tsz-Wo Nicholas Sze
6979cbfc1f HDFS-9838. Refactor the excessReplicateMap to a class. 2016-02-24 19:42:47 -08:00
Jing Zhao
47b92f2b6f HDFS-9837. BlockManager#countNodes should be able to detect duplicated internal blocks. Contributed by Jing Zhao. 2016-02-24 15:13:56 -08:00
cnauroth
d1dd248b75 HDFS-9854. Log cipher suite negotiation more verbosely. Contributed by Wei-Chiu Chuang. 2016-02-24 12:11:35 -08:00
Kihwal Lee
d27d7fc72e HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla. 2016-02-24 12:04:01 -06:00
Zhe Zhang
def754ec06 HDFS-9816. Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]. Contributed by Rui Li. 
Change-Id: I64b57bab4722cdc6e1e3148c3a3a401370249afe
 2016-02-23 14:05:17 -08:00
Colin Patrick Mccabe
211c78c090 HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via cmccabe) 2016-02-23 12:01:20 -08:00
Tsuyoshi Ozawa
48c61cd6aa HDFS-9084. Pagination, sorting and filtering of files/directories in the HDFS Web UI. (Contributed by Ravi Prakash) 2016-02-23 19:30:58 +09:00
Tsz-Wo Nicholas Sze
66289a3bf4 HDFS-8578. On upgrade, Datanode should process all storage/data dirs in parallel. Contributed by vinayakumarb and szetszwo 2016-02-22 15:01:15 -08:00
Zhe Zhang
a2fdfff02d HADOOP-12668. Support excluding weak Ciphers in HttpServer2 through ssl-server.conf. Contributed by Vijay Singh. 
Change-Id: Ie46a5427d29188935427f67e55203c19fcd83335
 2016-02-22 14:13:54 -08:00
Arpit Agarwal
a63301a864 HDFS-9842. dfs.datanode.balance.bandwidthPerSec should accept friendly size units. (Contributed by Lin Yiqun) 2016-02-22 11:34:09 -08:00
Vinayakumar B
342c9572bf HDFS-9425. Expose number of blocks per volume as a metric (Contributed by Brahma Reddy Battula) 2016-02-22 09:29:23 +05:30
Vinayakumar B
f313516731 HDFS-7452. skip StandbyException log for getCorruptFiles() (Contributed by Bramha Reddy Battula) 2016-02-22 09:21:46 +05:30
Arpit Agarwal
d5abd293a8 HDFS-9839. Reduce verbosity of processReport logging. (Contributed by Arpit Agarwal) 
This closes #78
 2016-02-20 23:19:09 -08:00
Jing Zhao
e54cc29312 HDFS-9818. Correctly handle EC reconstruction work caused by not enough racks. Contributed by Jing Zhao. 2016-02-19 19:02:23 -08:00
Jing Zhao
6546d9e7ff HDFS-9829. Erasure Coding: Improve few exception handling logic of ErasureCodingWorker. Contributed by Rakesh R. 2016-02-19 10:40:28 -08:00
Tsz-Wo Nicholas Sze
46f7790236 HDFS-8923. Add -source flag to balancer usage message. Contributed by Chris Trezzo 2016-02-18 18:18:12 -08:00
Kihwal Lee
ba6d5ed955 HDFS-4946. Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable. Contributed by James Kinley and Nathan Roberts. 2016-02-18 13:21:51 -06:00
cnauroth
5d1889a66d HDFS-9711. Integrate CSRF prevention filter in WebHDFS. Contributed by Chris Nauroth. 2016-02-18 10:07:28 -08:00
Akira Ajisaka
e78d8e66f7 HDFS-6832. Fix the usage of 'hdfs namenode' command. Contributed by Manjunath Ballur. 2016-02-19 01:35:06 +09:00
Andrew Wang
3a23dc683c HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. Contributed by Wei Zhou. 2016-02-17 11:29:58 -08:00
Haohui Mai
77f7ca3e94 HDFS-9815. Move o.a.h.fs.Hdfs to hadoop-hdfs-client. Contributed by Vinayakumar B. 2016-02-17 10:56:11 -08:00
Vinayakumar B
2536ece7b6 HDFS-9787. SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer changed to false. (Contributed by Guocui Mi) 2016-02-17 10:41:24 +05:30
Junping Du
77ba5add0d HDFS-9456. BlockPlacementPolicyWithNodeGroup should override verifyBlockPlacement(). Contributed by Xiaobing Zhou. 2016-02-16 18:55:55 -08:00
Arpit Agarwal
1de1641f17 HDFS-9801. ReconfigurableBase should update the cached configuration. (Arpit Agarwal) 2016-02-12 12:41:04 -08:00
Arun Suresh
9fdfb546fb HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level (Inigo Goiri via asuresh) 2016-02-12 11:18:10 -08:00
Jing Zhao
972782d956 HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. Contributed by Jing Zhao. 2016-02-12 11:07:52 -08:00
Akira Ajisaka
e6a7044b85 HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. Contributed by Lin Yiqun. 2016-02-13 01:57:24 +09:00
Arpit Agarwal
9b959f3ee7 HDFS-9790. HDFS Balancer should exit with a proper message if upgrade is not finalized (Contributed by Xiaobing Zhou) 2016-02-11 20:40:03 -08:00
Lei Xu
c7fcec24b8 HDFS-9795. OIV Delimited should show which files are ACL-enabled (lei) 2016-02-11 20:33:51 -08:00
Jing Zhao
19adb2bc64 HDFS-9789. Correctly update DataNode's scheduled block size when writing small EC file. Contributed by Jing Zhao. 2016-02-10 21:24:00 -08:00
Andrew Wang
aeb13ef2ab HDFS-9788. Incompatible tag renumbering in HeartbeatResponseProto. 2016-02-10 15:26:08 -08:00
Akira Ajisaka
802a7edfae HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-10 23:49:14 +09:00
Akira Ajisaka
5e565ce7a7 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta." 
This reverts commit fe124da5ff.
 2016-02-10 23:43:33 +09:00
Zhe Zhang
a0fb2eff9b HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to BlockReconstructionWork. Contributed by Rakesh R. 
Change-Id: I6dfc8efd94fa2bbb4eec0e4730a5a4f92c8a5519
 2016-02-09 14:43:59 -08:00
Allen Wittenauer
401ae4ecdb HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler (Ryan Sasson via aw) 2016-02-09 14:15:21 -08:00
zhezhang
dbe49c1bd6 HDFS-9244. Support nested encryption zones. 
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
 2016-02-08 16:30:51 -08:00
Uma Maheswara Rao G
bffaa38a91 HDFS-9777. Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh) 2016-02-08 13:47:45 -08:00
Kihwal Lee
193d27de0a HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su. 2016-02-08 12:16:29 -06:00
Uma Maheswara Rao G
aea31eee78 HDFS-9713. DataXceiver#copyBlock should return if block is pinned. Contributed by Uma Maheswara Rao G. 2016-02-07 20:50:47 -08:00
Akira Ajisaka
fe124da5ff HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-06 19:31:01 +09:00
cnauroth
c6497949e8 HDFS-9761. Rebalancer sleeps too long between iterations. Contributed by Mingliang Liu. 2016-02-05 15:07:43 -08:00
Tsz-Wo Nicholas Sze
4e5e1c0f99 HDFS-9726. Refactor IBR code to a new class. 2016-02-05 23:17:59 +08:00
Haohui Mai
1bcfab8e7f HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse ObjectMapper. Contributed by Akira Ajisaka. 2016-02-04 11:34:26 -08:00
Kihwal Lee
496f33de0c HDFS-9730. Storage ID update does not happen when there is a layout change. Contributed by Tsz Wo Nicholas Sze. 2016-02-04 13:19:54 -06:00
Vinayakumar B
a6728a013e HDFS-9629. Update the footer of Web UI to show year 2016 (Contributed by Xiao Chen and Vinayakumar B) 2016-02-04 13:37:51 +05:30
Jing Zhao
7badf15604 HDFS-9748. Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice. Contributed by Walter Su. 2016-02-03 13:30:09 -08:00
Vinayakumar B
04375756a5 HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed by Lei (Eddy) Xu) 2016-02-03 07:35:01 +05:30
zhezhang
4ae543fdcd HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to BlockECReconstructionCommand. Contributed by Rakesh R. 
Change-Id: I405365a8395770e494b92bfe9651f4f0366d8f28
 2016-02-02 12:32:08 -08:00
Colin Patrick Mccabe
dd9ebf6eed HDFS-9260. Improve the performance and GC friendliness of NameNode startup and full block reports (Staffan Friberg via cmccabe) 2016-02-02 11:23:00 -08:00
Colin Patrick Mccabe
2da03b48eb HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size (Elliot Clark via cmccabe) 2016-02-02 10:59:10 -08:00
Lei Xu
9d494f0c0e HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference. (Xiao Chen via lei) 2016-02-02 10:42:01 -08:00
Vinayakumar B
4d7055e0c8 HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys before initialize (Contributed by DENG FEI) 2016-02-02 12:51:35 +05:30
Yongjun Zhang
34ab50ea92 HDFS-9406. FSImage may get corrupted after deleting snapshot. (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang) 2016-02-01 13:56:55 -08:00
Lei Xu
e50aa53eed HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei) 2016-02-01 12:56:53 -08:00
Xiaoyu Yao
b11008d54d HDFS-9210. Fix some misuse of %n in VolumeScanner#printStats. Contributed by Xiaoyu Yao. 2016-02-01 11:55:27 -08:00
Steve Loughran
af2dccbca5 HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames (Mingliang Liu via stevel) 2016-02-01 16:01:50 +00:00
Vinayakumar B
8f2622b6a0 HDFS-9659. EditLogTailerThread to Active Namenode RPC should timeout (Contributed by surendra singh lilhore) 2016-02-01 14:10:55 +05:30
Vinayakumar B
e418bd1fb0 HDFS-9566. Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method (Contributed by Daryn Sharp) 2016-02-01 13:24:05 +05:30
Yongjun Zhang
8ee060311c HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. (Xiao Chen via Yongjun Zhang) 2016-01-28 23:04:03 -08:00
Colin Patrick Mccabe
f67149ab08 HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had an error (Rakesh R via cmccabe) 2016-01-28 19:54:50 -08:00
Tsz-Wo Nicholas Sze
662e17b46a HDFS-9654. Code refactoring for HDFS-8578. 2016-01-28 10:58:03 +08:00
Jing Zhao
ec25c7f9c7 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 16:34:40 -08:00
Jing Zhao
3a9571308e Revert "HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu." 
This reverts commit 8a91109d16.
 2016-01-27 16:31:19 -08:00
Jing Zhao
8a91109d16 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:48:47 -08:00
Tsz-Wo Nicholas Sze
45c763ad61 HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. 2016-01-26 11:20:13 +08:00
Tsz-Wo Nicholas Sze
bd909ed9f2 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-26 10:32:51 +08:00
Arpit Agarwal
d62b4a4de7 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:17:05 -08:00
Eric Yang
10a2bc0dff HDFS-9653. Added blocks pending deletion report to dfsadmin. 
(Weiwei Yang via eyang)
 2016-01-24 14:19:49 -08:00
Jing Zhao
95363bcc7d HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with length less than the first internal block. Contributed by Jing Zhao. 2016-01-22 09:46:02 -08:00
Kihwal Lee
2fd19b9674 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:10:06 -06:00
Masatake Iwasaki
ae9c61ff0a HDFS-9618. Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks (iwasakims) 2016-01-22 14:11:48 +09:00
Zhe Zhang
b4a05c1fd5 HDFS-9674. The HTrace span for OpWriteBlock should record the maxWriteToDisk time. Contributed by Colin McCabe. 
Change-Id: I9bf3f3bcd57f5880189ad7c160f3dd66f97d904b
 2016-01-21 13:25:42 -08:00
Jing Zhao
c304890c8c HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao. 2016-01-21 11:13:17 -08:00
Vinayakumar B
b7372b7166 HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI) 2016-01-21 19:17:05 +05:30