Wei-Chiu Chuang
8c591b8d19
HDFS-10974. Document replication factor for EC files. Contributed by Yiqun Lin.
2017-03-30 11:16:05 -07:00
Andrew Wang
0e6f8e4bc6
HDFS-10971. Distcp should not copy replication factor if source file is erasure coded. Contributed by Manoj Govindassamy.
2017-03-28 22:14:03 -07:00
Yongjun Zhang
d235dcdf0b
HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang)
2017-02-27 20:42:13 -08:00
Andrew Wang
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
Steve Loughran
ed33ce11dd
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
2017-01-19 11:25:40 +00:00
Chris Nauroth
4c8f9e1302
HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
2017-01-05 15:04:47 -08:00
Akira Ajisaka
209e805430
HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki.
2016-11-28 14:34:57 +09:00
Mingliang Liu
beb70fed4f
HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
...
This closes #131
2016-11-22 13:12:23 -08:00
Mingliang Liu
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
Masatake Iwasaki
0bdd263d82
HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
2016-10-27 15:46:59 +09:00
Yongjun Zhang
0f0c15f7a5
HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang.
2016-10-25 12:25:40 -07:00
Yongjun Zhang
3a60573039
Revert "Fix HDFS-11040"
...
This reverts commit 54c1815790
.
2016-10-25 12:25:02 -07:00
Yongjun Zhang
54c1815790
Fix HDFS-11040
2016-10-25 12:19:34 -07:00
Chris Douglas
a1a0281e12
HADOOP-13626. Remove distcp dependency on FileStatus serialization
2016-10-24 12:46:54 -07:00
Yongjun Zhang
8650cc84f2
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-19 17:37:54 -07:00
Xiao Chen
efdf810cf9
HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge.
2016-10-18 18:18:43 -07:00
Yongjun Zhang
0bc6d37f3c
Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang."
...
This reverts commit 412c4c9a34
.
2016-10-17 22:47:37 -07:00
Yongjun Zhang
412c4c9a34
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-17 11:04:42 -07:00
Jing Zhao
0a85d07983
HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin.
2016-10-13 13:24:54 -07:00
Brahma Reddy Battula
e17a4970be
HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore
2016-09-27 10:45:12 +05:30
Steve Loughran
e5ef51e717
HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri.
2016-09-23 10:01:30 +01:00
Chris Nauroth
98bdb51397
HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan.
2016-09-19 15:16:47 -07:00
Allen Wittenauer
58ed4fa544
HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS
...
This commit includes the following changes:
HADOOP-13356. Add a function to handle command_subcommand_OPTS
HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
HADOOP-13357. Modify common to use hadoop_subcommand_opts
HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
HADOOP-13564. modify mapred to use hadoop_subcommand_opts
HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
HADOOP-13360. Documentation for HADOOP_subcommand_OPTS
This closes apache/hadoop#126
2016-09-12 11:10:00 -07:00
Ravi Prakash
9faccd1046
HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic
2016-09-12 08:26:08 -07:00
Andrew Wang
da456ffd62
Preparing for 3.0.0-alpha2 development
2016-07-15 19:04:17 -07:00
Andrew Wang
f292624bd8
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
2016-07-11 18:06:28 -07:00
Yongjun Zhang
8113855b3a
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
2016-06-28 23:15:13 -07:00
Allen Wittenauer
422c73a865
HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw)
2016-06-28 07:21:04 -07:00
Yongjun Zhang
cfb860dee7
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
2016-05-26 23:30:31 -07:00
Steve Loughran
c918286b17
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:59 +01:00
Jing Zhao
03788d3015
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
2016-05-17 15:46:30 -07:00
Steve Loughran
c69a649257
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:00:18 +01:00
Allen Wittenauer
730bc746f9
HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)
...
This commit contains the following JIRA issues:
HADOOP-12931. bin/hadoop work for dynamic subcommands
HADOOP-12932. bin/yarn work for dynamic subcommands
HADOOP-12933. bin/hdfs work for dynamic subcommands
HADOOP-12934. bin/mapred work for dynamic subcommands
HADOOP-12935. API documentation for dynamic subcommands
HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
HADOOP-13086. enable daemonization of dynamic commands
HADOOP-13087. env var doc update for dynamic commands
HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
HADOOP-13089. hadoop distcp adds client opts twice when dynamic
HADOOP-13094. hadoop-common unit tests for dynamic commands
HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
HADOOP-13107. clean up how rumen is executed
HADOOP-13108. dynamic subcommands need a way to manipulate arguments
HADOOP-13110. add a streaming subcommand to mapred
HADOOP-13111. convert hadoop gridmix to be dynamic
HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
HADOOP-13117. clarify daemonization and security vars for dynamic commands
HADOOP-13120. add a --debug message when dynamic commands have been used
HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
HADOOP-13129. fix typo in dynamic subcommand docs
HADOOP-13151. Underscores should be escaped in dynamic subcommands document
HADOOP-13153. fix typo in debug statement for dynamic subcommands
2016-05-16 17:54:45 -07:00
Chris Nauroth
b9685e85d5
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
2016-05-16 11:53:17 -07:00
Andrew Wang
3c5c57af28
HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.
2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91
Revert "Update project version to 3.0.0-alpha1-SNAPSHOT."
...
This reverts commit 6b53802cba
.
2016-05-12 15:32:45 -07:00
Andrew Wang
6b53802cba
Update project version to 3.0.0-alpha1-SNAPSHOT.
2016-05-12 11:05:05 -07:00
Jing Zhao
af942585a1
HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu.
2016-05-04 10:23:04 -07:00
Yongjun Zhang
959a28dd12
HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang)
2016-04-26 16:08:03 -07:00
Akira Ajisaka
02c51c27d9
HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi.
2016-04-25 22:33:09 +09:00
Jing Zhao
63e5412f1a
HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou.
2016-04-22 15:14:40 -07:00
Yongjun Zhang
a749ba0cea
HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang)
2016-04-21 12:17:17 -07:00
Jing Zhao
404f57f328
HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi.
2016-04-14 10:35:22 -07:00
Akira Ajisaka
18c7e58283
HDFS-9640. Remove hsftp from DistCp in trunk. Contributed by Wei-Chiu Chuang.
2016-03-28 15:32:38 +09:00
Allen Wittenauer
738155063e
HADOOP-12857. rework hadoop-tools (aw)
2016-03-23 13:46:38 -07:00
Masatake Iwasaki
33a412e8a4
HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims)
2016-03-03 18:57:23 +09:00
Yongjun Zhang
ba1c9d484a
HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang)
2016-02-19 10:17:37 -08:00
Yongjun Zhang
eddd823cd6
HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)
2016-01-29 12:11:55 -08:00
Yongjun Zhang
a9c69ebeb7
HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang)
2016-01-15 10:03:09 -08:00
Colin Patrick Mccabe
8315582c4f
HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe)
2016-01-13 16:28:06 -08:00
Xiaoyu Yao
c2e2e13455
HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore.
2016-01-11 17:08:26 -08:00
Zhe Zhang
95f32015ad
HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng.
...
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
2016-01-11 09:46:56 -08:00
Akira Ajisaka
f94d892704
HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad.
2015-11-13 21:52:57 +09:00
Kihwal Lee
b24fe06483
Addendum to MAPREDUCE-6451
2015-10-30 16:05:23 -05:00
Kihwal Lee
2868ca0328
MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla.
2015-10-30 14:56:41 -05:00
Haohui Mai
da8441d0fe
HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu.
2015-10-13 15:24:24 -07:00
Haohui Mai
7e2c971fed
Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu."
...
This reverts commit fa5a54cac7
.
2015-10-10 03:24:59 -07:00
Haohui Mai
fa5a54cac7
HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu.
2015-10-09 22:38:06 -07:00
Harsh J
66dad854c0
MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian.
2015-09-28 13:12:43 +05:30
Tsuyoshi Ozawa
dfd807afab
HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang.
2015-09-22 12:54:29 +09:00
Steve Loughran
7269906254
HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel).
2015-09-13 14:25:26 +01:00
Yongjun Zhang
0bc15cb6e6
HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang)
2015-08-20 08:02:54 -07:00
Robert Kanter
aaafa0b2ee
HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter)
2015-06-30 16:49:27 -07:00
Amareshwari Sriramadasu
8ef07f767f
HADOOP-11203. Allow ditscp to accept bandwitdh in fraction MegaBytes. Contributed by Raju Bairishetti
2015-06-26 09:52:06 +05:30
Colin Patrick Mccabe
7dba7005b7
HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe)
2015-05-26 12:15:46 -07:00
Jing Zhao
0790275f05
HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase.
2015-05-18 13:24:35 -07:00
Jonathan Eagles
2d7363b273
HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles)
2015-05-01 13:11:50 -05:00
Ravi Prakash
a287d2fb77
HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak)
2015-04-24 13:39:07 -07:00
Ravi Prakash
cfba355052
HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak)
2015-04-21 16:49:37 -07:00
Jing Zhao
4c097e473b
HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao.
2015-04-15 12:37:20 -07:00
Colin Patrick Mccabe
932730df7d
HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe)
2015-04-03 14:08:25 -07:00
Haohui Mai
75cb1d42ab
HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao.
2015-04-01 16:50:59 -07:00
Tsuyoshi Ozawa
e074952bd6
HADOOP-11760. Fix typo of javadoc in DistCp. Contributed by Brahma Reddy Battula.
2015-03-27 23:15:51 +09:00
Yongjun Zhang
44809b8081
HADOOP-11724. DistCp throws NPE when the target directory is root. (Lei Eddy Xu via Yongjun Zhang)
2015-03-25 15:45:45 -07:00
Tsuyoshi Ozawa
ef9946cd52
HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal tags in hadoop-tools. Contributed by Akira AJISAKA.
2015-03-17 16:09:21 +09:00
Jing Zhao
ed70fa142c
HDFS-7535. Utilize Snapshot diff report for distcp. Contributed by Jing Zhao.
2015-03-04 10:30:53 -08:00
Vinod Kumar Vavilapalli
5af693fde2
MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public API in DistCp. Contributed by Jing Zhao.
2015-03-03 16:28:41 -08:00
Tsuyoshi Ozawa
9ae7f9eb7b
MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA.
2015-03-03 18:06:26 +09:00
Tsuyoshi Ozawa
d1c6accb6f
HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
2015-03-03 14:17:52 +09:00
Allen Wittenauer
039366e3b4
MAPREDUCE-5653. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw)
2015-02-28 22:53:38 -08:00
Tsuyoshi Ozawa
9cedad11d8
Revert "HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)"
...
This reverts commit 946456c6d8
.
Conflicts:
hadoop-common-project/hadoop-common/CHANGES.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
2015-02-25 00:32:04 +09:00
Tsuyoshi Ozawa
946456c6d8
HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
2015-02-19 13:06:53 +09:00
Allen Wittenauer
7776847a59
HADOOP-11437. Remove the version and author information from distcp's README file (Brahma Reddy Battula via aw)
2015-02-11 15:47:36 -08:00
Allen Wittenauer
9112f093cd
HADOOP-10976. moving the source code of hadoop-tools docs to the directory under hadoop-tools (Masatake Iwasaki via aw)
2015-02-04 17:57:34 -08:00
Haohui Mai
2e98ad34ce
HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack. Contributed by Li Lu.
2014-12-09 20:45:21 -08:00
Jing Zhao
57cb43be50
HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. Contributed by Ted Yu.
2014-12-08 11:08:39 -08:00
Allen Wittenauer
f614cb71d4
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw)
2014-09-25 11:27:33 -07:00
Allen Wittenauer
3cde37c991
HADOOP-11009. Add Timestamp Preservation to DistCp (Gary Steelman via aw)
2014-09-24 15:38:30 -07:00
Charles Lamb
14b01dd046
HDFS-6872. Fix TestOptionsParser. (clamb)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/fs-encryption@1618974 13f79535-47bb-0310-9956-ffa450edef68
2014-08-19 20:41:15 +00:00
Charles Lamb
041b8326a1
MAPREDUCE-6007. Add support to distcp to preserve raw.* namespace extended attributes. (clamb)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/fs-encryption@1616657 13f79535-47bb-0310-9956-ffa450edef68
2014-08-08 01:32:18 +00:00
Andrew Wang
e8a1ff7efe
MAPREDUCE-5971. Move the default options for distcp -p to DistCpOptionSwitch. Contributed by Charles Lamb.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611217 13f79535-47bb-0310-9956-ffa450edef68
2014-07-16 23:42:34 +00:00
Andrew Wang
50cd2a6bed
HADOOP-10666. Remove Copyright /d/d/d/d Apache Software Foundation from the source files license header. Contributed by Henry Saputra.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603025 13f79535-47bb-0310-9956-ffa450edef68
2014-06-16 23:23:33 +00:00
Uma Maheswara Rao G
f81c7b0252
MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs). Contributed by Yi Liu.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600900 13f79535-47bb-0310-9956-ffa450edef68
2014-06-06 14:45:39 +00:00
Jing Zhao
3671a5e16f
MAPREDUCE-5899. Support incremental data copy in DistCp. Contributed by Jing Zhao.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1596931 13f79535-47bb-0310-9956-ffa450edef68
2014-05-22 18:17:11 +00:00
Chris Nauroth
11be7334c4
MAPREDUCE-5809. Enhance distcp to support preserving HDFS ACLs. Contributed by Chris Nauroth.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1595283 13f79535-47bb-0310-9956-ffa450edef68
2014-05-16 18:25:23 +00:00
Tsz-wo Sze
03db13206f
MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE, MAX_CHUNKS_IDEAL, MIN_RECORDS_PER_CHUNK and SPLIT_RATIO to be configurable. Contributed by Tsuyoshi OZAWA
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1592703 13f79535-47bb-0310-9956-ffa450edef68
2014-05-06 10:24:10 +00:00
Jonathan Turner Eagles
a059eadbe9
MAPREDUCE-5639. Port DistCp2 document to trunk (Akira AJISAKA via jeagles)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1590058 13f79535-47bb-0310-9956-ffa450edef68
2014-04-25 15:23:42 +00:00
Chris Nauroth
06c15b6a3e
MAPREDUCE-5852. Prepare MapReduce codebase for JUnit 4.11. Contributed by Chris Nauroth.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1589006 13f79535-47bb-0310-9956-ffa450edef68
2014-04-21 23:44:41 +00:00
Aaron Myers
640a097533
HADOOP-10459. distcp V2 doesn't preserve root dir's attributes when -p is specified. Contributed by Yongjun Zhang.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1584227 13f79535-47bb-0310-9956-ffa450edef68
2014-04-03 00:32:25 +00:00
Haohui Mai
fb1d7fb596
HDFS-5570. Deprecate hftp / hsftp and replace them with webhdfs / swebhdfs. Contributed by Haohui Mai.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1584100 13f79535-47bb-0310-9956-ffa450edef68
2014-04-02 17:28:08 +00:00