big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,713 Commits 2 Branches 2 Tags 264 MiB
f6367c5f44
Commit Graph

4107 Commits

Author SHA1 Message Date
Kihwal Lee
5f34402ada HDFS-10779. Rename does not need to re-solve destination. Contributed by Daryn Sharp. 2016-09-28 17:57:23 -05:00
Anu Engineer
03f519a757 HDFS-9850. DiskBalancer: Explore removing references to FsVolumeSpi. Contributed by Manoj Govindassamy. 2016-09-27 21:35:06 -07:00
Arpit Agarwal
8ae4729107 HDFS-10828. Fix usage of FsDatasetImpl object lock in ReplicaMap. (Arpit Agarwal) 2016-09-27 10:02:15 -07:00
Arpit Agarwal
059058f961 HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha Koneru. 2016-09-26 17:09:47 -07:00
Wei-Chiu Chuang
3ae652f821 HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. Contributed by Wei-Chiu Chuang. 2016-09-26 13:11:32 -07:00
Brahma Reddy Battula
3e37e243ee HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by Jagadesh Kiran N 2016-09-25 11:02:46 +05:30
Wei-Chiu Chuang
74b3dd514c HDFS-10876. Dispatcher#dispatch should log IOException stacktrace. Contributed by Manoj Govindassamy. 2016-09-23 13:26:57 -07:00
Konstantin V Shvachko
a5bb88c8e0 HDFS-10843. Update space quota when a UC block is completed rather than committed. Contributed by Erik Krogen. 2016-09-23 10:37:46 -07:00
Anu Engineer
d85d9b2e7b HDFS-10871. DiskBalancerWorkItem should not import jackson relocated by htrace. Contributed by Manoj Govindassamy. 2016-09-22 19:36:16 -07:00
Andrew Wang
40acacee08 HDFS-10877. Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers. Contributed by Sean Mackrory. 2016-09-22 11:43:11 -07:00
Xiao Chen
e52d6e7a46 HDFS-10875. Optimize du -x to cache intermediate result. Contributed by Xiao Chen. 2016-09-19 21:44:42 -07:00
Andrew Wang
c54f6ef30f HDFS-10868. Remove stray references to DFS_HDFS_BLOCKS_METADATA_ENABLED. 2016-09-19 11:17:03 -07:00
Xiao Chen
ea839bd48e HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen. 2016-09-17 22:25:39 -07:00
Akira Ajisaka
b09a03cd7d HDFS-10862. Typos in 4 log messages. Contributed by Mehran Hassani. 2016-09-16 16:08:47 +09:00
Arpit Agarwal
ec3ea18875 HDFS-9895. Remove unnecessary conf cache from DataNode. Contributed by Xiaobing Zhou. 2016-09-15 15:25:47 -07:00
Kai Zheng
ea0c2b8b05 HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng 2016-09-15 10:46:00 +08:00
Lei Xu
86c9862bec HDFS-10636. Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File. (Virajith Jalaparti via lei) 2016-09-13 12:54:14 -07:00
Andrew Wang
db6d243cf8 HDFS-10837. Standardize serializiation of WebHDFS DirectoryListing. 2016-09-13 11:02:36 -07:00
Anu Engineer
e3f7f58a5f HDFS-10599. DiskBalancer: Execute CLI via Shell. Contributed by Manoj Govindassamy. 2016-09-13 09:38:12 -07:00
Akira Ajisaka
f0876b8b60 HDFS-10856. Update the comment of BPServiceActor$Scheduler#scheduleNextBlockReport. Contributed by Yiqun Lin. 2016-09-13 14:50:03 +09:00
Jing Zhao
72dfb048a9 HDFS-10858. FBR processing may generate incorrect reportedBlock-blockGroup mapping. Contributed by Jing Zhao. 2016-09-12 16:40:11 -07:00
Anu Engineer
8a93f45a80 HDFS-10821. DiskBalancer: Report command support with multiple nodes. Contributed by Yiqun Lin. 2016-09-12 15:45:26 -07:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS 
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
 2016-09-12 11:10:00 -07:00
Xiao Chen
cc01ed7026 HDFS-10855. Fix typos in HDFS documents. Contributed by Yiqun Lin. 2016-09-10 23:24:49 -07:00
Arpit Agarwal
a99bf26a08 HDFS-10830. FsDatasetImpl#removeVolumes crashes with IllegalMonitorStateException when vol being removed is in use. (Arpit Agarwal and Manoj Govindassamy) 2016-09-10 18:22:25 -07:00
Anu Engineer
bee9f57f5c HDFS-10808. DiskBalancer does not execute multi-steps plan-redux. Contributed by Anu Engineer. 2016-09-09 15:00:39 -07:00
Anu Engineer
baab48922a HDFS-9849. DiskBalancer: reduce lock path in shutdown code. Contributed by Yuanbo Liu. 2016-09-08 20:00:42 -07:00
Anu Engineer
35c5943b8b HDFS-10553. DiskBalancer: Rename Tools/DiskBalancer class to Tools/DiskBalancerCLI. Contributed by Manoj Govindassamy. 2016-09-08 19:26:56 -07:00
Chris Douglas
011f3b24d4 HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang 2016-09-08 17:53:40 -07:00
Chris Douglas
b6d839a60c HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin 2016-09-08 13:29:30 -07:00
Akira Ajisaka
63f594892e HDFS-10778. Add -format option to make the output of FileDistribution processor human-readable in OfflineImageViewer. 2016-09-08 15:13:43 +09:00
Kai Zheng
d355573f56 Revert "HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng" 
This reverts commit 62a9667136
 2016-09-08 05:50:17 +08:00
Arpit Agarwal
5f23abfa30 HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:37:21 -07:00
Xiao Chen
f6c0b7543f HDFS-10841. Remove duplicate or unused variable in appendFile(). Contributed by Kihwal Lee. 2016-09-06 11:48:35 -07:00
Chris Nauroth
f0d5382ff3 HDFS-6962. ACL inheritance conflicts with umaskmode. Contributed by Chris Nauroth. 2016-09-06 11:02:39 -07:00
Chris Douglas
d37dc5d1b8 HDFS-9847. HDFS configuration should accept time units. Contributed by Yiqun Lin 2016-09-06 10:38:04 -07:00
Kai Zheng
62a9667136 HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng 2016-09-07 17:05:33 +08:00
Xiao Chen
07650bc37a HDFS-9781. FsDatasetImpl#getBlockReports can occasionally throw NullPointerException. Contributed by Manoj Govindassamy. 2016-09-02 15:33:11 -07:00
Zhe Zhang
6f4b0d33ca HDFS-10817. Add Logging for Long-held NN Read Locks. Contributed by Erik Krogen. 2016-08-31 15:40:09 -07:00
Andrew Wang
85bab5fb57 HDFS-10784. Implement WebHdfsFileSystem#listStatusIterator. 2016-08-31 14:29:37 -07:00
Kihwal Lee
01721dd88e HDFS-10729. Improve log message for edit loading failures caused by FS limit checks. Contributed by Wei-Chiu Chuang. 2016-08-31 14:02:37 -05:00
Anu Engineer
20ae1fa259 HDFS-10813. DiskBalancer: Add the getNodeList method in Command. Contributed by Yiqun Lin. 2016-08-30 18:42:55 -07:00
Ming Ma
9dcbdbdb5a HDFS-9392. Admins support for maintenance state. Contributed by Ming Ma. 2016-08-30 14:00:13 -07:00
Wei-Chiu Chuang
c4ee6915a1 HDFS-10760. DataXceiver#run() should not log InvalidToken exception as an error. Contributed by Pan Yuxuan. 2016-08-30 10:43:20 -07:00
Xiao Chen
cd5e10ccca HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge. 2016-08-29 17:48:08 -07:00
Yongjun Zhang
5d1609ddf2 HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin. 2016-08-29 13:59:54 -07:00
Yongjun Zhang
c25817159a HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang. 2016-08-27 22:51:31 -07:00
Zhe Zhang
e806db7190 HDFS-10584. Allow long-running Mover tool to login with keytab. Contributed by Rakesh R. 2016-08-26 16:43:45 -07:00
Andrew Wang
a445b82baa HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy. 2016-08-26 15:39:18 -07:00
Zhe Zhang
407b519fb1 HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 2016-08-26 14:19:55 -07:00
Kihwal Lee
8b7adf4ddf HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp. 2016-08-26 15:39:56 -05:00
Kihwal Lee
a1f3293762 HDFS-10772. Reduce byte/string conversions for get listing. Contributed by Daryn Sharp. 2016-08-24 15:21:08 -05:00
Daryn Sharp
ec252ce0fc HDFS-10762. Pass IIP for file status related methods 2016-08-24 08:46:47 -05:00
Wei-Chiu Chuang
f0efea490e HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-23 04:14:24 -07:00
Akira Ajisaka
e90f3359de HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 2016-08-23 19:57:23 +09:00
Eric Yang
c49333becf HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) 2016-08-22 18:29:56 -07:00
Kihwal Lee
3ca4d6ddfd Revert "HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp." 
This reverts commit 22fc46d765.
 2016-08-22 16:57:45 -05:00
Kihwal Lee
22fc46d765 HDFS-10762. Pass IIP for file status related methods. Contributed by Daryn Sharp. 2016-08-22 15:37:02 -05:00
Jing Zhao
0faee62a0c HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 2016-08-19 22:13:36 -07:00
Kihwal Lee
2550371f66 HDFS-10711. Optimize FSPermissionChecker group membership check. Contributed by Daryn Sharp. 2016-08-19 09:12:17 -05:00
Akira Ajisaka
8179f9a493 HDFS-10645. Make block report size as a metric and add this metric to datanode web ui. Contributed by Yuanbo Liu. 2016-08-19 16:15:00 +09:00
Arpit Agarwal
ca13e7971d HDFS-10773. BlockSender should not synchronize on the dataset object. (Contributed by Chen Liang) 2016-08-17 16:29:08 -07:00
Kihwal Lee
869393643d HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 2016-08-17 15:53:03 -05:00
Anu Engineer
b047bc7270 HDFS-10560. DiskBalancer: Reuse ObjectMapper instance to improve the performance. Contributed by Yiqun Lin. 2016-08-16 10:20:08 -07:00
Mingliang Liu
ef55fe1716 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:23:47 -07:00
Anu Engineer
5628b36c08 HDFS-10559. DiskBalancer: Use SHA1 for Plan ID. Contributed by Xiaobing Zhou. 2016-08-15 20:10:21 -07:00
Anu Engineer
02abd131b8 HDFS-10567. Improve plan command help message. Contributed by Xiaobing Zhou. 2016-08-15 19:58:57 -07:00
Kihwal Lee
864f878d59 HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 2016-08-15 17:28:09 -05:00
Kihwal Lee
03dea65e0b HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 2016-08-15 16:45:44 -05:00
Anu Engineer
bed69d18e6 HDFS-10580. DiskBalancer: Make use of unused methods in GreedyPlanner to print debug info. Contributed by Yiqun Lin 2016-08-15 12:40:29 -07:00
Kihwal Lee
83e57e083f HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 2016-08-15 13:01:23 -05:00
Anu Engineer
9f29f423e4 HDFS-10737. disk balancer add volume path to report command. Contributed by Yuanbo Liu. 2016-08-15 09:47:30 -07:00
Wei-Chiu Chuang
9019606b69 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 2016-08-12 05:58:13 -07:00
Kai Zheng
b5af9be72c HDFS-8668. Erasure Coding: revisit buffer used for encoding and decoding. Contributed by Sammi Chen 2016-08-13 13:52:37 +08:00
Wei-Chiu Chuang
a428d4f50e HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 2016-08-10 22:17:35 -07:00
Xiaoyu Yao
ec289bbece HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:49:36 -07:00
Xiaoyu Yao
fdafd0cb4f Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca7.
 2016-08-10 15:47:55 -07:00
Xiaoyu Yao
9f816d7ca7 Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 2016-08-10 15:14:38 -07:00
Yuanbo Liu
10e84c6a6e HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 2016-08-10 11:07:42 -07:00
Wei-Chiu Chuang
d00d3add9e HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-09 18:11:34 -07:00
Lei Xu
9c6a4383ca HDFS-10681. DiskBalancer: query command should report Plan file path apart from PlanID. (Manoj Govindassamy via lei) 2016-08-09 15:59:22 -07:00
Lei Xu
cc48251bfd HDFS-10457. DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) 2016-08-09 15:51:19 -07:00
Kihwal Lee
b10c936020 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 2016-08-09 11:56:21 -05:00
Arpit Agarwal
8c0638471f HADOOP-10682. Replace FsDatasetImpl object lock with a separate lock object. (Chen Liang) 2016-08-08 12:02:53 -07:00
Konstantin V Shvachko
4d3af47f27 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 2016-08-07 14:29:26 -07:00
Akira Ajisaka
ef432579a7 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 2016-08-05 17:31:11 +09:00
Wei-Chiu Chuang
18d9e6ec0b HDFS-10588. False alarm in datanode log - ERROR - Disk Balancer is not enabled. Contributed by Weiwei Yang. 2016-08-04 19:40:33 -07:00
Kihwal Lee
438a9f047e HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 2016-08-04 16:14:13 -05:00
Kihwal Lee
57369a678c HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 2016-08-04 14:55:21 -05:00
Tsz-Wo Nicholas Sze
8c9b44eaff In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:45:40 -07:00
Kihwal Lee
6ae39199da HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 2016-08-04 09:07:34 -05:00
Kihwal Lee
6f63566694 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 2016-08-03 15:20:20 -05:00
Jing Zhao
f4ba5ff1d7 HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 2016-08-03 12:42:05 -07:00
Kihwal Lee
22ef5286bc HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 2016-08-03 13:12:43 -05:00
Kihwal Lee
58db263e93 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 2016-08-03 12:14:59 -05:00
Kihwal Lee
bebf10d245 HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 2016-08-03 11:53:41 -05:00
Akira Ajisaka
a5fb298e56 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-02 17:07:59 +09:00
Konstantin V Shvachko
c4463f2ef2 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 85a20508bd.
 2016-08-01 22:34:40 -07:00
Daryn Sharp
9f473cf903 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 2016-08-01 10:14:28 -05:00
Yongjun Zhang
8d32bd884d HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 2016-07-29 23:20:14 -07:00