big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,002 Commits 2 Branches 2 Tags 264 MiB
9fcc3a1fc8
Commit Graph

780 Commits

Author SHA1 Message Date
John Zhuge
5a40bafdae HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge. 
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
 2017-03-13 08:11:25 -07:00
Steve Loughran
092ec39fb9
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials. 
Contributed by Yuanbo Liu
 2017-03-10 17:43:22 +00:00
Mingliang Liu
881ec4d97b HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu 2017-03-10 00:16:09 -08:00
John Zhuge
c5ee7fded4 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge. 
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
 2017-03-09 18:30:17 -08:00
John Zhuge
98142d2f72 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran" 
This reverts commit 53a12fa721.
 2017-03-08 09:54:22 -08:00
Mingliang Liu
28daaf0eb2 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak 2017-03-07 14:55:52 -08:00
Robert Kanter
1598fd3b79 YARN-6275. Fail to show real-time tracking charts in SLS (yufeigu via rkanter) 2017-03-07 13:47:52 -08:00
Akira Ajisaka
f597f4c43e
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 2017-03-07 15:14:55 +09:00
Mingliang Liu
686823529b HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-06 17:16:36 -08:00
Mingliang Liu
52d7d5aa1a Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth" 
This reverts commit 6b7cd62b8c.
 2017-03-06 17:10:11 -08:00
Mingliang Liu
c571cda5c7 HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA 2017-03-06 17:00:13 -08:00
Mingliang Liu
6b7cd62b8c HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:12 -08:00
Akira Ajisaka
490abfb10f HADOOP-14057. Fix package.html to compile with Java 9. 2017-03-04 00:25:22 +09:00
Steve Loughran
a97833e0ed
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38bf40373d2855163fa6444821b0f994c)
 2017-03-02 21:44:30 +00:00
Allen Wittenauer
989bd56b9f HADOOP-12556. KafkaSink jar files are created but not copied to target dist (Babak Behzad via aw) 2017-02-28 16:15:40 -08:00
Wangda Tan
eac6b4c35c YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 2017-02-28 10:35:50 -08:00
Akira Ajisaka
258342e76c HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. 2017-02-28 13:47:44 +09:00
Yongjun Zhang
d235dcdf0b HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 2017-02-27 20:42:13 -08:00
Steve Loughran
dab00da19f
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.

(cherry picked from commit 29fe5af017b945d8750c074ca39031b5b777eddd)
 2017-02-25 15:35:19 +00:00
Sunil G
b32ffa2753 YARN-4779. Fix AM container allocation logic in SLS. Contributed by Wangda Tan. 2017-02-24 21:39:25 +05:30
Steve Loughran
e60c6543d5
HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran 2017-02-24 13:24:59 +00:00
Steve Loughran
9c22a91662
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 
(cherry picked from commit ff87ca84418a710c6dc884fe8c70947fcc6489d5)
 2017-02-24 10:41:36 +00:00
Mingliang Liu
82607fce39 HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari. 2017-02-23 13:48:44 -08:00
Mingliang Liu
a4d4a23785 HADOOP-13321. Deprecate FileSystem APIs that promote inefficient call patterns. Contributed by Chris Nauroth and Mingliang Liu 2017-02-23 12:55:40 -08:00
Mingliang Liu
0013090fb4 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 2017-02-22 13:34:20 -08:00
Steve Loughran
4f4250fbcc HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:43:48 +00:00
Mingliang Liu
924def7854 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane 2017-02-21 13:44:42 -08:00
Steve Loughran
2158496f6b HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
 2017-02-21 18:28:49 +00:00
Steve Loughran
8035749c26 HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:46 +00:00
Steve Loughran
3a2e30fa9f HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1dc997906390d653dfafb4f0d7e193c4)
 2017-02-18 18:17:11 +00:00
Mingliang Liu
658702efff HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 2017-02-16 16:51:03 -08:00
Mingliang Liu
bdad8b7b97 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 2017-02-16 16:41:31 -08:00
Mingliang Liu
f432999025 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane 2017-02-16 15:14:25 -08:00
Kai Zheng
cd3e59a3dc HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu 2017-02-15 16:34:30 +08:00
Akira Ajisaka
b9f8491252 HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 2017-02-15 01:45:56 +09:00
Lei Xu
71c23c9fc9 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei) 2017-02-13 13:33:13 -08:00
Kai Zheng
01be4503c3 HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui 2017-02-13 13:18:20 +08:00
Lei Xu
839b690ed5 HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 13:59:03 -08:00
Arpit Agarwal
2b7a7bbe0f HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus. 2017-02-10 10:09:31 -08:00
Kai Zheng
5b151290ae HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation deleteDirs. Contributed by Genmao Yu 2017-02-10 14:53:18 +08:00
Kai Zheng
a8a594b4c8 HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui 2017-02-09 18:00:22 +08:00
Akira Ajisaka
98c35bb1a4 HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows. 2017-01-21 17:31:20 +09:00
Lei Xu
ccf2d66244 HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) 2017-01-21 10:52:40 +08:00
Lei Xu
e015b56319 HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 2017-01-20 14:48:04 +08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Steve Loughran
ed33ce11dd HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
 2017-01-19 11:25:40 +00:00
Steve Loughran
e648b6e138 HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:25:15 +00:00
Wangda Tan
287d3d6804 YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 2017-01-09 08:40:39 -08:00
Chris Nauroth
4c8f9e1302 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 2017-01-05 15:04:47 -08:00
Wangda Tan
0a55bd841e YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 2017-01-05 10:31:05 -08:00
Wangda Tan
5ed63e3e9d YARN-4899. Queue metrics of SLS capacity scheduler only activated after app submit to the queue. (Jonathan Hung via wangda) 2017-01-04 19:17:11 -08:00
Akira Ajisaka
01d31fe938 HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 2017-01-04 14:10:36 +09:00
Mingliang Liu
c6a5b689db HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki 2016-12-29 16:24:06 -08:00
Mingliang Liu
e92a77099b HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 2016-12-22 20:15:18 -08:00
Akira Ajisaka
ae401539ea YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 2016-12-22 14:32:24 +09:00
Wangda Tan
3bcfe3a9de YARN-5706. Fail to launch SLSRunner due to NPE. (Kai Sasaki via wangda) 2016-12-20 11:03:29 -08:00
Mingliang Liu
f92913c35b HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade 2016-12-15 12:35:08 -08:00
Mingliang Liu
ef34bf2bb9 HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane 2016-12-13 10:34:32 -08:00
Mingliang Liu
c6a3923245 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 2016-12-12 14:55:34 -08:00
Mingliang Liu
15dd1f3381 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth 2016-12-05 12:04:07 -08:00
Mingliang Liu
4113ec5fa5 HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane 2016-12-02 15:54:57 -08:00
Mingliang Liu
51211a7d7a HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 2016-12-02 14:01:42 -08:00
Mingliang Liu
2ff84a0040 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 2016-12-02 13:36:04 -08:00
Mingliang Liu
3fd844b99f HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 2016-11-30 13:01:19 -08:00
Mingliang Liu
d60a60be8a HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 2016-11-28 16:30:29 -08:00
Akira Ajisaka
209e805430 HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki. 2016-11-28 14:34:57 +09:00
Akira Ajisaka
9e173d2bac HADOOP-13816. Ambiguous plugin version warning from maven build. Contributed by Kai Sasaki. 2016-11-28 13:10:31 +09:00
Steve Loughran
0de0c32ddd HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:39 +00:00
Mingliang Liu
af791b774b HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:00:35 -08:00
Mingliang Liu
beb70fed4f HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 
This closes #131
 2016-11-22 13:12:23 -08:00
Wei-Chiu Chuang
c0b1a44f6c HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean Mackrory. 2016-11-17 22:48:35 -06:00
Mingliang Liu
5af572b644 HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 2016-11-15 10:57:00 -08:00
Chris Douglas
5c61ad2488 HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane 2016-11-11 11:15:07 -08:00
Mingliang Liu
89354f0475 HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth 2016-11-10 08:58:37 -08:00
Jason Lowe
3f93ac0733 YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 2016-11-08 22:01:26 +00:00
Xiao Chen
acd509dc57 HADOOP-13795. Skip testGlobStatusThrowsExceptionForUnreadableDir in TestFSMainOperationsSwift. Contributed by John Zhuge. 2016-11-07 09:21:01 -08:00
Akira Ajisaka
b970446b2c HADOOP-13798. TestHadoopArchives times out. 2016-11-07 19:53:43 +09:00
Akira Ajisaka
de0132748d MAPREDUCE-6796. Remove unused properties from JTConfig.java. Contributed by Haibo Chen. 2016-11-05 03:49:19 +09:00
Arun Suresh
0aafc122d4 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 2016-11-04 07:31:54 -07:00
Xiao Chen
20c4d8efab HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. Contributed by John Zhuge. 2016-11-03 11:16:07 -07:00
Steve Loughran
a1761a841e HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:54:46 +00:00
Robert Kanter
5877f20f9c HADOOP-10075. Update jetty dependency to version 9 (rkanter) 2016-10-27 16:09:00 -07:00
Masatake Iwasaki
0bdd263d82 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 2016-10-27 15:46:59 +09:00
Chris Nauroth
1f8490a5ba HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 2016-10-26 08:55:42 -07:00
Chris Nauroth
9cad3e2350 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 2016-10-26 08:27:26 -07:00
Akira Ajisaka
d3bb69a667 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 2016-10-26 15:32:07 +09:00
Yongjun Zhang
0f0c15f7a5 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 2016-10-25 12:25:40 -07:00
Yongjun Zhang
3a60573039 Revert "Fix HDFS-11040" 
This reverts commit 54c1815790.
 2016-10-25 12:25:02 -07:00
Yongjun Zhang
54c1815790 Fix HDFS-11040 2016-10-25 12:19:34 -07:00
Chris Nauroth
309a43925c HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 2016-10-25 09:03:03 -07:00
Chris Nauroth
3372e94030 HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 2016-10-24 21:54:06 -07:00
Chris Nauroth
d8fa1cfa67 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 2016-10-24 21:22:34 -07:00
Chris Douglas
a1a0281e12 HADOOP-13626. Remove distcp dependency on FileStatus serialization 2016-10-24 12:46:54 -07:00
Anu Engineer
3fbf4cd5da HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:33:58 -07:00
Steve Loughran
9ae270af02 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:59 +01:00
Yongjun Zhang
8650cc84f2 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-19 17:37:54 -07:00
Xiao Chen
efdf810cf9 HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge. 2016-10-18 18:18:43 -07:00
Steve Loughran
6c348c5691 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 21:16:02 +01:00
Yongjun Zhang
0bc6d37f3c Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang." 
This reverts commit 412c4c9a34.
 2016-10-17 22:47:37 -07:00
Andrew Wang
987ee51141 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 2016-10-17 13:25:58 -07:00
Yongjun Zhang
412c4c9a34 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 2016-10-17 11:04:42 -07:00
Mingliang Liu
d9f73f1b7c HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu 2016-10-13 22:33:55 -07:00
Jing Zhao
0a85d07983 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 2016-10-13 13:24:54 -07:00
Kai Zheng
c31b5e61b1 HADOOP-13701. AbstractContractRootDirectoryTest can fail when handling delete "/". Contributed by Genmao Yu 2016-10-08 17:35:59 +06:00
Kai Zheng
a57bba470b Merge branch 'trunk' into HADOOP-12756 2016-10-09 10:29:40 +08:00
Chris Nauroth
69620f9559 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 2016-10-07 11:41:19 -07:00
Steve Loughran
ebd4f39a39 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:51:40 +01:00
Steve Loughran
2cc841f16e HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka. 2016-10-06 16:31:00 +01:00
Steve Loughran
d6be1e75d8 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:02:11 +01:00
Akira Ajisaka
6cfe649375 HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. Contributed by Grant Henke. 
This closes #133
 2016-10-05 20:44:39 +09:00
Chris Nauroth
88b9444a81 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 2016-10-04 10:36:58 -07:00
Steve Loughran
ee0c722dc8 HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:01:00 +01:00
Chris Nauroth
47f80922dc HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 2016-09-28 16:19:32 -07:00
Kai Zheng
d1443988f8 Revert "HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back" 
This reverts commit 5707f88d85
 2016-09-29 09:18:27 +08:00
Brahma Reddy Battula
e17a4970be HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 2016-09-27 10:45:12 +05:30
Kai Zheng
5707f88d85 HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back. 
HADOOP-12756 branch: Incorporate Aliyun OSS file system implementation. Contributors:
Mingfei Shi (mingfei.shi@intel.com)
Genmao Yu (genmao.ygm@alibaba-inc.com)
 2016-09-26 20:42:22 +08:00
Steve Loughran
e5ef51e717 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:01:30 +01:00
Kai Zheng
26d5df390c HADOOP-13634. Some configuration in doc has been outdated. Contributed by Genmao Yu 2016-09-23 08:44:28 +08:00
Kai Zheng
a49b3be38e Merge branch 'trunk' into HADOOP-12756 2016-09-21 14:03:21 +08:00
Kai Zheng
22af6f8db3 HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu 2016-09-21 14:02:44 +08:00
Kai Zheng
08b37603d9 HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu 2016-09-20 15:12:02 +08:00
Kai Zheng
9cd4760257 HADOOP-13609. Refine credential provider related codes for AliyunOss integration. Contributed by Genmao Yu 2016-09-20 11:39:02 +08:00
Kai Zheng
005f4528c7 Merge branch 'trunk' into HADOOP-12756 2016-09-20 08:09:29 +08:00
Chris Nauroth
98bdb51397 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 2016-09-19 15:16:47 -07:00
Kai Zheng
a1940464a4 HADOOP-13610. Clean up AliyunOss integration tests. Contributed by Genmao Yu 2016-09-18 19:10:48 +08:00
Kai Zheng
dbb28eb1f2 Merge branch 'trunk' into HADOOP-12756 2016-09-18 19:03:39 +08:00
Mingliang Liu
96142efa2d HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu. 2016-09-17 22:07:46 -07:00
Steve Loughran
53a12fa721 HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:45:04 +01:00
Kai Zheng
60f66a9306 Merge branch 'trunk' into HADOOP-12756 2016-09-14 16:12:40 +08:00
Allen Wittenauer
58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS 
This commit includes the following changes:

	HADOOP-13356. Add a function to handle command_subcommand_OPTS
	HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
	HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
	HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
	HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
	HADOOP-13357. Modify common to use hadoop_subcommand_opts
	HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
	HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
	HADOOP-13564. modify mapred to use hadoop_subcommand_opts
	HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
	HADOOP-13360. Documentation for HADOOP_subcommand_OPTS

This closes apache/hadoop#126
 2016-09-12 11:10:00 -07:00
Ravi Prakash
9faccd1046 HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic 2016-09-12 08:26:08 -07:00
Steve Loughran
4b6d795f28 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:55:32 +01:00
Steve Loughran
7fdfcd8a6c HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:25:23 +01:00
Mingfei
e671a0f52b HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu. 2016-09-07 11:17:43 +08:00
Mingfei
d33e928fbe HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
cdb77110e7 HADOOP-13498. The number of multi-part upload part should not bigger than 10000. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
6bb741b9f8 HADOOP-13499. Support session credentials for authenticating with Aliyun. Contributed by Genmao Yu. 2016-09-07 11:15:48 +08:00
Mingfei
bd2d97adee HADOOP-13483. File create should throw error rather than overwrite directories. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Mingfei
4d84c814fc HADOOP-13491. Fix several warnings from findbugs. Contributed by Genmao Yu. 2016-09-07 11:15:47 +08:00
Kai Zheng
a5d5342228 HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou 2016-09-07 11:15:47 +08:00
Chris Nauroth
d152557cf7 HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 2016-09-06 09:36:21 -07:00
Arun Suresh
d6d9cff21b YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 2016-08-30 15:52:29 -07:00
Xiaoyu Yao
8b57be108f HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth. 
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
(cherry picked from commit a65e159c59de41d26e12212dd1b7c28c28265cc3)
 2016-08-29 15:30:49 -07:00
Chris Douglas
c37346d0e3 HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton. 2016-08-23 14:12:52 -07:00
Chris Nauroth
6f9c346e57 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:18:49 -07:00
Chris Nauroth
763f0497bb HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 2016-08-19 10:48:10 -07:00
Chris Nauroth
ae4db25443 HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 2016-08-18 10:44:36 -07:00
Junping Du
0da69c324d YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 
(cherry picked from commit d464483bf7f0b3e3be3ba32cd6c3eee546747ab5)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
 2016-08-18 07:27:23 -07:00
Steve Loughran
040c185d62 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:55 +01:00
Chris Nauroth
822d661b8f HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 2016-08-17 14:54:54 -07:00
Chris Nauroth
3808876c73 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 2016-08-16 17:05:52 -07:00
Chris Nauroth
2ed58c40e5 HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka. 2016-08-08 12:28:40 -07:00
Chris Nauroth
4ef1bc0bdb HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth. 2016-08-02 22:32:21 -07:00
Chris Nauroth
b43de80031 HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg. 2016-07-27 15:50:38 -07:00
Akira Ajisaka
54fe17a607 HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák. 2016-07-27 19:41:09 +09:00
Steve Loughran
86ae218893 HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:50:23 +01:00
Steve Loughran
be9e46b42d HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:45:03 +01:00
Steve Loughran
7052ca8804 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 18:00:08 +01:00
Chris Nauroth
c60a68ce1d HADOOP-13392. [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Contributed by Vishwajeet Dusane and Chris Douglas. 2016-07-22 09:16:38 -07:00
Chris Douglas
dca6335dbc HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS documentation 2016-07-21 23:21:47 -07:00
mattf-horton
12aa184479 HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 14:43:57 -07:00
Steve Loughran
37362c2f92 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:51 +01:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Jitendra Pandey
c2f9cd584c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:28:53 -07:00
Jian He
819224dcf9 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:36:20 -07:00
Andrew Wang
f292624bd8 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 2016-07-11 18:06:28 -07:00
Jitendra Pandey
6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Jitendra Pandey
aa42c7a6dd HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:34:03 -07:00
Steve Loughran
96fa0f848b HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu. 2016-06-29 16:04:50 +01:00
Akira Ajisaka
8d202f1258 HADOOP-13235. Use Date and Time API in KafkaSink. 2016-06-29 07:40:12 -07:00
Yongjun Zhang
8113855b3a HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 2016-06-28 23:15:13 -07:00
Allen Wittenauer
422c73a865 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 2016-06-28 07:21:04 -07:00
Steve Loughran
4ee3543625 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:45:25 +01:00
Chris Nauroth
b2c596cdda HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 2016-06-21 11:28:52 -07:00
Jitendra Pandey
b7c4cf7129 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:25:30 -07:00
Chris Nauroth
51d16e7b38 HADOOP-13242. Authenticate to Azure Data Lake using client ID and keys. Contributed by Atul Sikaria. 2016-06-16 23:35:20 -07:00
Chris Nauroth
127d2c7281 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 2016-06-16 11:18:02 -07:00
Ravi Prakash
4aefe119a0 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 11:13:35 -07:00
Chris Nauroth
c9e71382a5 HADOOP-12875. [Azure Data Lake] Support for contract test and unit test cases. Contributed by Vishwajeet Dusane. 2016-06-16 09:46:05 -07:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Chris Nauroth
9581fb715c HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane. 2016-06-09 14:33:31 -07:00
Andrew Wang
58c3486850 HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris Douglas. 2016-06-09 13:49:52 -07:00
Steve Loughran
31ffaf76f2 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 21:00:47 +01:00
Steve Loughran
656c460c0e HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 17:28:49 +01:00
Steve Loughran
a3f78d8fa8 HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:42:36 +02:00
Chris Nauroth
c58a59f708 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:55:33 -07:00
Chris Nauroth
16b1cc7af9 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 2016-06-01 14:49:22 -07:00
Steve Loughran
5870611030 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:18:20 +01:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Yongjun Zhang
cfb860dee7 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 2016-05-26 23:30:31 -07:00
Steve Loughran
39ec1515a2 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 16:39:31 +01:00
Steve Loughran
757050ff35 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:15 +01:00
Steve Loughran
c918286b17 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:59 +01:00
Chris Nauroth
93258459fa HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 2016-05-19 22:00:21 -07:00
Jing Zhao
03788d3015 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 2016-05-17 15:46:30 -07:00
Steve Loughran
08ea07f1b8 HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:19:29 +01:00
Steve Loughran
c69a649257 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:00:18 +01:00
Allen Wittenauer
730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) 
This commit contains the following JIRA issues:

    HADOOP-12931. bin/hadoop work for dynamic subcommands
    HADOOP-12932. bin/yarn work for dynamic subcommands
    HADOOP-12933. bin/hdfs work for dynamic subcommands
    HADOOP-12934. bin/mapred work for dynamic subcommands
    HADOOP-12935. API documentation for dynamic subcommands
    HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
    HADOOP-13086. enable daemonization of dynamic commands
    HADOOP-13087. env var doc update for dynamic commands
    HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
    HADOOP-13089. hadoop distcp adds client opts twice when dynamic
    HADOOP-13094. hadoop-common unit tests for dynamic commands
    HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
    HADOOP-13107. clean up how rumen is executed
    HADOOP-13108. dynamic subcommands need a way to manipulate arguments
    HADOOP-13110. add a streaming subcommand to mapred
    HADOOP-13111. convert hadoop gridmix to be dynamic
    HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
    HADOOP-13117. clarify daemonization and security vars for dynamic commands
    HADOOP-13120. add a --debug message when dynamic commands have been used
    HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
    HADOOP-13129. fix typo in dynamic subcommand docs
    HADOOP-13151. Underscores should be escaped in dynamic subcommands document
    HADOOP-13153. fix typo in debug statement for dynamic subcommands
 2016-05-16 17:54:45 -07:00
Chris Nauroth
b9685e85d5 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 2016-05-16 11:53:17 -07:00
Steve Loughran
dd3a8bed0a HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:47:12 +01:00