big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,054 Commits 2 Branches 2 Tags 264 MiB
025c656572
Commit Graph

153 Commits

Author SHA1 Message Date
Steve Loughran
f36cbc8475 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.
 2017-10-20 22:27:04 +01:00
ChenSammi
e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
Xiaoyu Yao
63720ef574 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-05 23:34:55 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Andrew Wang
f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Andrew Wang
dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Sean Mackrory
1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Wei-Chiu Chuang
44350fdf49 HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default). Contributed by Bharat Viswanadham. 2017-07-20 18:23:13 -07:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Jason Lowe
dd65eea74b HADOOP-8143. Change distcp to have -pb on by default. Contributed by Mithun Radhakrishnan 2017-06-20 09:53:47 -05:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Sunil G
b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Yongjun Zhang
b4adc8392c HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-18 15:35:22 -07:00
Mingliang Liu
26172a94d6 HADOOP-14267. Make DistCpOptions immutable. Contributed by Mingliang Liu 2017-03-31 20:04:26 -07:00
Yongjun Zhang
bf3fb585aa HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-03-30 17:38:56 -07:00
Yongjun Zhang
144f1cf765 Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen." 
This reverts commit 064c8b25ec.
 2017-03-30 17:38:18 -07:00
Yongjun Zhang
064c8b25ec HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen. 2017-03-30 17:01:15 -07:00
Wei-Chiu Chuang
8c591b8d19 HDFS-10974. Document replication factor for EC files. Contributed by Yiqun Lin. 2017-03-30 11:16:05 -07:00
Andrew Wang
0e6f8e4bc6 HDFS-10971. Distcp should not copy replication factor if source file is erasure coded. Contributed by Manoj Govindassamy. 2017-03-28 22:14:03 -07:00
Yongjun Zhang
d235dcdf0b HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 2017-02-27 20:42:13 -08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Steve Loughran
ed33ce11dd HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
 2017-01-19 11:25:40 +00:00
Chris Nauroth
4c8f9e1302 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 2017-01-05 15:04:47 -08:00
Akira Ajisaka
209e805430 HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki. 2016-11-28 14:34:57 +09:00
Mingliang Liu
beb70fed4f HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 
This closes #131
 2016-11-22 13:12:23 -08:00
Mingliang Liu
5af572b644 HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 2016-11-15 10:57:00 -08:00
Masatake Iwasaki
0bdd263d82 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 2016-10-27 15:46:59 +09:00
Yongjun Zhang
0f0c15f7a5 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 2016-10-25 12:25:40 -07:00
Yongjun Zhang
3a60573039 Revert "Fix HDFS-11040" 
This reverts commit 54c1815790.
 2016-10-25 12:25:02 -07:00
Yongjun Zhang
54c1815790 Fix HDFS-11040 2016-10-25 12:19:34 -07:00
Chris Douglas
a1a0281e12 HADOOP-13626. Remove distcp dependency on FileStatus serialization 2016-10-24 12:46:54 -07:00
Yongjun Zhang
8650cc84f2 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-19 17:37:54 -07:00
Xiao Chen
efdf810cf9 HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge. 2016-10-18 18:18:43 -07:00
Yongjun Zhang
0bc6d37f3c Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang." 
This reverts commit 412c4c9a34.
 2016-10-17 22:47:37 -07:00
Yongjun Zhang
412c4c9a34 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-17 11:04:42 -07:00
Jing Zhao
0a85d07983 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 2016-10-13 13:24:54 -07:00
Brahma Reddy Battula
e17a4970be HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 2016-09-27 10:45:12 +05:30
Steve Loughran
e5ef51e717 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:01:30 +01:00
Chris Nauroth
98bdb51397 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 2016-09-19 15:16:47 -07:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS 
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
 2016-09-12 11:10:00 -07:00
Ravi Prakash
9faccd1046 HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic 2016-09-12 08:26:08 -07:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Andrew Wang
f292624bd8 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 2016-07-11 18:06:28 -07:00
Yongjun Zhang
8113855b3a HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 2016-06-28 23:15:13 -07:00
Allen Wittenauer
422c73a865 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 2016-06-28 07:21:04 -07:00
Yongjun Zhang
cfb860dee7 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 2016-05-26 23:30:31 -07:00
Steve Loughran
c918286b17 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:59 +01:00
Jing Zhao
03788d3015 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 2016-05-17 15:46:30 -07:00
Steve Loughran
c69a649257 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:00:18 +01:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00