big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
14,710 Commits 2 Branches 2 Tags 264 MiB
dbd205762e
Commit Graph

540 Commits

Author SHA1 Message Date
Chris Nauroth
3372e94030 HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 2016-10-24 21:54:06 -07:00
Chris Nauroth
d8fa1cfa67 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 2016-10-24 21:22:34 -07:00
Chris Douglas
a1a0281e12 HADOOP-13626. Remove distcp dependency on FileStatus serialization 2016-10-24 12:46:54 -07:00
Anu Engineer
3fbf4cd5da HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:33:58 -07:00
Steve Loughran
9ae270af02 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:59 +01:00
Yongjun Zhang
8650cc84f2 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-19 17:37:54 -07:00
Xiao Chen
efdf810cf9 HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge. 2016-10-18 18:18:43 -07:00
Steve Loughran
6c348c5691 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 21:16:02 +01:00
Yongjun Zhang
0bc6d37f3c Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang." 
This reverts commit 412c4c9a34.
 2016-10-17 22:47:37 -07:00
Andrew Wang
987ee51141 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 2016-10-17 13:25:58 -07:00
Yongjun Zhang
412c4c9a34 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-17 11:04:42 -07:00
Mingliang Liu
d9f73f1b7c HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu 2016-10-13 22:33:55 -07:00
Jing Zhao
0a85d07983 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 2016-10-13 13:24:54 -07:00
Kai Zheng
c31b5e61b1 HADOOP-13701. AbstractContractRootDirectoryTest can fail when handling delete "/". Contributed by Genmao Yu 2016-10-08 17:35:59 +06:00
Kai Zheng
a57bba470b Merge branch 'trunk' into HADOOP-12756 2016-10-09 10:29:40 +08:00
Chris Nauroth
69620f9559 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 2016-10-07 11:41:19 -07:00
Steve Loughran
ebd4f39a39 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:51:40 +01:00
Steve Loughran
2cc841f16e HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka. 2016-10-06 16:31:00 +01:00
Steve Loughran
d6be1e75d8 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:02:11 +01:00
Akira Ajisaka
6cfe649375 HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. Contributed by Grant Henke. 
This closes #133
 2016-10-05 20:44:39 +09:00
Chris Nauroth
88b9444a81 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 2016-10-04 10:36:58 -07:00
Steve Loughran
ee0c722dc8 HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:01:00 +01:00
Chris Nauroth
47f80922dc HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 2016-09-28 16:19:32 -07:00
Kai Zheng
d1443988f8 Revert "HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back" 
This reverts commit 5707f88d85
 2016-09-29 09:18:27 +08:00
Brahma Reddy Battula
e17a4970be HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 2016-09-27 10:45:12 +05:30
Kai Zheng
5707f88d85 HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back. 
HADOOP-12756 branch: Incorporate Aliyun OSS file system implementation. Contributors:
Mingfei Shi (mingfei.shi@intel.com)
Genmao Yu (genmao.ygm@alibaba-inc.com)
 2016-09-26 20:42:22 +08:00
Steve Loughran
e5ef51e717 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:01:30 +01:00
Kai Zheng
26d5df390c HADOOP-13634. Some configuration in doc has been outdated. Contributed by Genmao Yu 2016-09-23 08:44:28 +08:00
Kai Zheng
a49b3be38e Merge branch 'trunk' into HADOOP-12756 2016-09-21 14:03:21 +08:00
Kai Zheng
22af6f8db3 HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu 2016-09-21 14:02:44 +08:00
Kai Zheng
08b37603d9 HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu 2016-09-20 15:12:02 +08:00
Kai Zheng
9cd4760257 HADOOP-13609. Refine credential provider related codes for AliyunOss integration. Contributed by Genmao Yu 2016-09-20 11:39:02 +08:00
Kai Zheng
005f4528c7 Merge branch 'trunk' into HADOOP-12756 2016-09-20 08:09:29 +08:00
Chris Nauroth
98bdb51397 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 2016-09-19 15:16:47 -07:00
Kai Zheng
a1940464a4 HADOOP-13610. Clean up AliyunOss integration tests. Contributed by Genmao Yu 2016-09-18 19:10:48 +08:00
Kai Zheng
dbb28eb1f2 Merge branch 'trunk' into HADOOP-12756 2016-09-18 19:03:39 +08:00
Mingliang Liu
96142efa2d HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu. 2016-09-17 22:07:46 -07:00
Steve Loughran
53a12fa721 HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:45:04 +01:00
Kai Zheng
60f66a9306 Merge branch 'trunk' into HADOOP-12756 2016-09-14 16:12:40 +08:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS 
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
 2016-09-12 11:10:00 -07:00
Ravi Prakash
9faccd1046 HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic 2016-09-12 08:26:08 -07:00
Steve Loughran
4b6d795f28 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:55:32 +01:00
Steve Loughran
7fdfcd8a6c HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:25:23 +01:00
Mingfei
e671a0f52b HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu. 2016-09-07 11:17:43 +08:00
Mingfei
d33e928fbe HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
cdb77110e7 HADOOP-13498. The number of multi-part upload part should not bigger than 10000. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
6bb741b9f8 HADOOP-13499. Support session credentials for authenticating with Aliyun. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
bd2d97adee HADOOP-13483. File create should throw error rather than overwrite directories. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Mingfei
4d84c814fc HADOOP-13491. Fix several warnings from findbugs. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Kai Zheng
a5d5342228 HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou 2016-09-07 11:15:47 +08:00