HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop. Contributed by Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, Alexander Stojanovic, Brian Swan, and Min Wei.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601781 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Chris Nauroth
parent
c802ca28fd
commit
81bc395deb
@ -370,6 +370,11 @@ Release 2.5.0 - UNRELEASED
|
||||
|
||||
HADOOP-9704. Write metrics sink plugin for Hadoop/Graphite (Chu Tong, Alex Newman and Babak Behzad via raviprak)
|
||||
|
||||
HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop.
|
||||
(Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao,
|
||||
Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys,
|
||||
Alexander Stojanovic, Brian Swan, and Min Wei via cnauroth)
|
||||
|
||||
IMPROVEMENTS
|
||||
|
||||
HADOOP-10451. Remove unused field and imports from SaslRpcServer.
|
||||
|
||||
@ -309,6 +309,12 @@
|
||||
<artifactId>hadoop-openstack</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-azure</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
@ -768,6 +774,12 @@
|
||||
<version>1.8</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.microsoft.windowsazure.storage</groupId>
|
||||
<artifactId>microsoft-windowsazure-storage-sdk</artifactId>
|
||||
<version>0.6.0</version>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
|
||||
1
hadoop-tools/hadoop-azure/.gitignore
vendored
Normal file
1
hadoop-tools/hadoop-azure/.gitignore
vendored
Normal file
@ -0,0 +1 @@
|
||||
.checkstyle
|
||||
104
hadoop-tools/hadoop-azure/README.txt
Normal file
104
hadoop-tools/hadoop-azure/README.txt
Normal file
@ -0,0 +1,104 @@
|
||||
=============
|
||||
Building
|
||||
=============
|
||||
basic compilation:
|
||||
> mvn clean compile test-compile
|
||||
|
||||
Compile, run tests and produce jar
|
||||
> mvn clean package
|
||||
|
||||
=============
|
||||
Unit tests
|
||||
=============
|
||||
Most of the tests will run without additional configuration.
|
||||
For complete testing, configuration in src/test/resources is required:
|
||||
src/test/resources/azure-test.xml
|
||||
src/test/resources/log4j.properties
|
||||
|
||||
From command-line
|
||||
------------------
|
||||
Basic execution:
|
||||
> mvn test
|
||||
|
||||
NOTES:
|
||||
- The mvn pom.xml includes src/test/resources in the runtime classpath
|
||||
- detailed output (such as log4j) appears in target\surefire-reports\TEST-{testName}.xml
|
||||
including log4j messages.
|
||||
|
||||
Run the tests and generate report:
|
||||
> mvn site (at least once to setup some basics including images for the report)
|
||||
> mvn surefire-report:report (run and produce report)
|
||||
> mvn mvn surefire-report:report-only (produce report from last run)
|
||||
> mvn mvn surefire-report:report-only -DshowSuccess=false (produce report from last run, only show errors)
|
||||
> .\target\site\surefire-report.html (view the report)
|
||||
|
||||
Via eclipse
|
||||
-------------
|
||||
Manually add src\test\resources to the classpath for test run configuration:
|
||||
- run menu|run configurations|{configuration}|classpath|User Entries|advanced|add folder
|
||||
|
||||
Then run via junit test runner.
|
||||
NOTE:
|
||||
- if you change log4.properties, rebuild the project to refresh the eclipse cache.
|
||||
|
||||
Run Tests against Mocked storage.
|
||||
---------------------------------
|
||||
These run automatically and make use of an in-memory emulation of azure storage.
|
||||
|
||||
|
||||
Running tests against the Azure storage emulator
|
||||
---------------------------------------------------
|
||||
A selection of tests can run against the Azure Storage Emulator which is
|
||||
a high-fidelity emulation of live Azure Storage. The emulator is sufficient for high-confidence testing.
|
||||
The emulator is a Windows executable that runs on a local machine.
|
||||
|
||||
To use the emulator, install Azure SDK 2.3 and start the storage emulator
|
||||
See http://msdn.microsoft.com/en-us/library/azure/hh403989.aspx
|
||||
|
||||
Enable the Azure emulator tests by setting
|
||||
fs.azure.test.emulator -> true
|
||||
in src\test\resources\azure-test.xml
|
||||
|
||||
Running tests against live Azure storage
|
||||
-------------------------------------------------------------------------
|
||||
In order to run WASB unit tests against a live Azure Storage account, add credentials to
|
||||
src\test\resources\azure-test.xml. These settings augment the hadoop configuration object.
|
||||
|
||||
For live tests, set the following in azure-test.xml:
|
||||
1. "fs.azure.test.account.name -> {azureStorageAccountName}
|
||||
2. "fs.azure.account.key.{AccountName} -> {fullStorageKey}"
|
||||
|
||||
=============
|
||||
Findbugs
|
||||
=============
|
||||
Run findbugs and show interactive GUI for review of problems
|
||||
> mvn findbugs:gui
|
||||
|
||||
Run findbugs and fail build if errors are found:
|
||||
> mvn findbugs:check
|
||||
|
||||
For help with findbugs plugin.
|
||||
> mvn findbugs:help
|
||||
|
||||
=============
|
||||
Checkstyle
|
||||
=============
|
||||
Rules for checkstyle @ src\config\checkstyle.xml
|
||||
- these are based on a core set of standards, with exclusions for non-serious issues
|
||||
- as a general plan it would be good to turn on more rules over time.
|
||||
- Occasionally, run checkstyle with the default Sun rules by editing pom.xml.
|
||||
|
||||
Command-line:
|
||||
> mvn checkstyle:check --> just test & fail build if violations found
|
||||
> mvn site checkstyle:checkstyle --> produce html report
|
||||
> . target\site\checkstyle.html --> view report.
|
||||
|
||||
Eclipse:
|
||||
- add the checkstyle plugin: Help|Install, site=http://eclipse-cs.sf.net/update
|
||||
- window|preferences|checkstyle. Add src/config/checkstyle.xml. Set as default.
|
||||
- project|properties|create configurations as required, eg src/main/java -> src/config/checkstyle.xml
|
||||
|
||||
NOTE:
|
||||
- After any change to the checkstyle rules xml, use window|preferences|checkstyle|{refresh}|OK
|
||||
|
||||
|
||||
19
hadoop-tools/hadoop-azure/dev-support/findbugs-exclude.xml
Normal file
19
hadoop-tools/hadoop-azure/dev-support/findbugs-exclude.xml
Normal file
@ -0,0 +1,19 @@
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
contributor license agreements. See the NOTICE file distributed with
|
||||
this work for additional information regarding copyright ownership.
|
||||
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
(the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
<FindBugsFilter>
|
||||
|
||||
</FindBugsFilter>
|
||||
202
hadoop-tools/hadoop-azure/pom.xml
Normal file
202
hadoop-tools/hadoop-azure/pom.xml
Normal file
@ -0,0 +1,202 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License. See accompanying LICENSE file.
|
||||
-->
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-project</artifactId>
|
||||
<version>3.0.0-SNAPSHOT</version>
|
||||
<relativePath>../../hadoop-project</relativePath>
|
||||
</parent>
|
||||
<artifactId>hadoop-azure</artifactId>
|
||||
<name>Apache Hadoop Azure support</name>
|
||||
<description>
|
||||
This module contains code to support integration with Azure.
|
||||
Currently this consists of a filesystem client to read data from
|
||||
and write data to Azure Storage.
|
||||
</description>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<properties>
|
||||
<file.encoding>UTF-8</file.encoding>
|
||||
<downloadSources>true</downloadSources>
|
||||
</properties>
|
||||
|
||||
<build>
|
||||
|
||||
<testResources>
|
||||
<testResource>
|
||||
<directory>src/test/resources</directory>
|
||||
<includes>
|
||||
<include>log4j.properties</include>
|
||||
</includes>
|
||||
</testResource>
|
||||
<testResource>
|
||||
<directory>src/test/resources</directory>
|
||||
<includes>
|
||||
<include>azure-test.xml</include>
|
||||
</includes>
|
||||
</testResource>
|
||||
</testResources>
|
||||
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.codehaus.mojo</groupId>
|
||||
<artifactId>findbugs-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<findbugsXmlOutput>true</findbugsXmlOutput>
|
||||
<xmlOutput>true</xmlOutput>
|
||||
<excludeFilterFile>${basedir}/dev-support/findbugs-exclude.xml
|
||||
</excludeFilterFile>
|
||||
<effort>Max</effort>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-site-plugin</artifactId>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.apache.maven.doxia</groupId>
|
||||
<artifactId>doxia-module-markdown</artifactId>
|
||||
<version>1.3</version>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
<configuration>
|
||||
<inputEncoding>UTF-8</inputEncoding>
|
||||
<outputEncoding>UTF-8</outputEncoding>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-project-info-reports-plugin</artifactId>
|
||||
|
||||
<configuration>
|
||||
<dependencyDetailsEnabled>false</dependencyDetailsEnabled>
|
||||
<dependencyLocationsEnabled>false</dependencyLocationsEnabled>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-checkstyle-plugin</artifactId>
|
||||
<configuration>
|
||||
<!-- To run with the default Sun ruleset,
|
||||
comment out the configLocation line -->
|
||||
<configLocation>src/config/checkstyle.xml</configLocation>
|
||||
</configuration>
|
||||
|
||||
</plugin>
|
||||
</plugins>
|
||||
|
||||
|
||||
<!--
|
||||
The following is to suppress a m2e warning in eclipse
|
||||
(m2e doesn't know how to handle maven-enforcer:enforce, so we have to tell m2e to ignore it)
|
||||
see: http://stackoverflow.com/questions/13040788/how-to-elimate-the-maven-enforcer-plugin-goal-enforce-is-ignored-by-m2e-wa
|
||||
-->
|
||||
<pluginManagement>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.eclipse.m2e</groupId>
|
||||
<artifactId>lifecycle-mapping</artifactId>
|
||||
<version>1.0.0</version>
|
||||
<configuration>
|
||||
<lifecycleMappingMetadata>
|
||||
<pluginExecutions>
|
||||
<pluginExecution>
|
||||
<pluginExecutionFilter>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-enforcer-plugin</artifactId>
|
||||
<versionRange>[1.0.0,)</versionRange>
|
||||
<goals>
|
||||
<goal>enforce</goal>
|
||||
</goals>
|
||||
</pluginExecutionFilter>
|
||||
<action>
|
||||
<ignore />
|
||||
</action>
|
||||
</pluginExecution>
|
||||
</pluginExecutions>
|
||||
</lifecycleMappingMetadata>
|
||||
</configuration>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</pluginManagement>
|
||||
|
||||
</build>
|
||||
|
||||
<!-- see hadoop-project/pom.xml for version number declarations -->
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-common</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.codehaus.jackson</groupId>
|
||||
<artifactId>jackson-mapper-asl</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.codehaus.jackson</groupId>
|
||||
<artifactId>jackson-core-asl</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.httpcomponents</groupId>
|
||||
<artifactId>httpclient</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.microsoft.windowsazure.storage</groupId>
|
||||
<artifactId>microsoft-windowsazure-storage-sdk</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.google.guava</groupId>
|
||||
<artifactId>guava</artifactId>
|
||||
<scope>compile</scope>
|
||||
</dependency>
|
||||
|
||||
|
||||
<!-- dependencies use for test only -->
|
||||
<dependency>
|
||||
<groupId>commons-io</groupId>
|
||||
<artifactId>commons-io</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-common</artifactId>
|
||||
<scope>test</scope>
|
||||
<type>test-jar</type>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
</project>
|
||||
184
hadoop-tools/hadoop-azure/src/config/checkstyle.xml
Normal file
184
hadoop-tools/hadoop-azure/src/config/checkstyle.xml
Normal file
@ -0,0 +1,184 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE module PUBLIC
|
||||
"-//Puppy Crawl//DTD Check Configuration 1.2//EN"
|
||||
"http://www.puppycrawl.com/dtds/configuration_1_2.dtd">
|
||||
|
||||
<!--
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
<!--
|
||||
Checkstyle configuration that checks the sun coding conventions from:
|
||||
|
||||
- the Java Language Specification at
|
||||
http://java.sun.com/docs/books/jls/second_edition/html/index.html
|
||||
|
||||
- the Sun Code Conventions at http://java.sun.com/docs/codeconv/
|
||||
|
||||
- the Javadoc guidelines at
|
||||
http://java.sun.com/j2se/javadoc/writingdoccomments/index.html
|
||||
|
||||
- the JDK Api documentation http://java.sun.com/j2se/docs/api/index.html
|
||||
|
||||
- some best practices
|
||||
|
||||
Checkstyle is very configurable. Be sure to read the documentation at
|
||||
http://checkstyle.sf.net (or in your downloaded distribution).
|
||||
|
||||
Most Checks are configurable, be sure to consult the documentation.
|
||||
To completely disable a check, just comment it out or delete it from the file.
|
||||
Finally, it is worth reading the documentation.
|
||||
-->
|
||||
|
||||
<module name="Checker">
|
||||
<!-- Checks that each Java package has a Javadoc file used for commenting. -->
|
||||
<!-- See http://checkstyle.sf.net/config_javadoc.html#JavadocPackage -->
|
||||
<module name="JavadocPackage">
|
||||
<property name="allowLegacy" value="true"/>
|
||||
</module>
|
||||
|
||||
<!-- Checks that property files contain the same keys. -->
|
||||
<!-- See http://checkstyle.sf.net/config_misc.html#Translation -->
|
||||
<module name="Translation"/>
|
||||
|
||||
<module name="FileLength">
|
||||
<property name="max" value="3000"/>
|
||||
</module>
|
||||
|
||||
<module name="FileTabCharacter">
|
||||
<property name="eachLine" value="true"/>
|
||||
</module>
|
||||
|
||||
<module name="TreeWalker">
|
||||
|
||||
<property name="cacheFile" value="${checkstyle.cache.file}"/>
|
||||
|
||||
<!-- Checks for Javadoc comments. -->
|
||||
<!-- See http://checkstyle.sf.net/config_javadoc.html -->
|
||||
|
||||
<module name="JavadocType">
|
||||
<property name="scope" value="public"/>
|
||||
</module>
|
||||
|
||||
<!--
|
||||
<module name="JavadocMethod"/>
|
||||
<module name="JavadocVariable"/>
|
||||
<module name="JavadocStyle"/>
|
||||
-->
|
||||
|
||||
|
||||
<!-- Checks for Naming Conventions. -->
|
||||
<!-- See http://checkstyle.sf.net/config_naming.html -->
|
||||
<module name="ConstantName"/>
|
||||
<module name="LocalFinalVariableName"/>
|
||||
<module name="LocalVariableName"/>
|
||||
<module name="MemberName"/>
|
||||
<module name="MethodName"/>
|
||||
<module name="PackageName"/>
|
||||
<module name="ParameterName"/>
|
||||
<module name="StaticVariableName"/>
|
||||
<module name="TypeName"/>
|
||||
|
||||
<!-- Checks for imports -->
|
||||
<!-- See http://checkstyle.sf.net/config_import.html -->
|
||||
<module name="AvoidStarImport"/>
|
||||
<module name="IllegalImport"/> <!-- defaults to sun.* packages -->
|
||||
<module name="RedundantImport"/>
|
||||
<module name="UnusedImports"/>
|
||||
|
||||
|
||||
<!-- Checks for Size Violations. -->
|
||||
<!-- See http://checkstyle.sf.net/config_sizes.html -->
|
||||
|
||||
<module name="LineLength">
|
||||
<property name="max" value="160" />
|
||||
</module>
|
||||
|
||||
<module name="MethodLength">
|
||||
<property name="max" value="3000"/>
|
||||
</module>
|
||||
|
||||
<module name="ParameterNumber"/>
|
||||
|
||||
|
||||
<!-- Checks for whitespace -->
|
||||
<!-- See http://checkstyle.sf.net/config_whitespace.html -->
|
||||
|
||||
<module name="EmptyForIteratorPad"/>
|
||||
<module name="MethodParamPad"/>
|
||||
<!-- module name="NoWhitespaceAfter"/> -->
|
||||
<module name="NoWhitespaceBefore"/>
|
||||
<module name="OperatorWrap"/>
|
||||
<module name="ParenPad"/>
|
||||
<module name="TypecastParenPad"/>
|
||||
<module name="WhitespaceAfter"/>
|
||||
<!-- <module name="WhitespaceAround"/> -->
|
||||
|
||||
|
||||
<!-- Modifier Checks -->
|
||||
<!-- See http://checkstyle.sf.net/config_modifiers.html -->
|
||||
<module name="ModifierOrder"/>
|
||||
<module name="RedundantModifier"/>
|
||||
|
||||
|
||||
<!-- Checks for blocks. You know, those {}'s -->
|
||||
<!-- See http://checkstyle.sf.net/config_blocks.html -->
|
||||
<module name="AvoidNestedBlocks"/>
|
||||
<!-- <module name="EmptyBlock"/> -->
|
||||
<module name="LeftCurly"/>
|
||||
<module name="NeedBraces"/>
|
||||
|
||||
<!-- <module name="RightCurly"/> -->
|
||||
|
||||
|
||||
<!-- Checks for common coding problems -->
|
||||
<!-- See http://checkstyle.sf.net/config_coding.html -->
|
||||
<!-- <module name="AvoidInlineConditionals"/> -->
|
||||
<!-- <module name="DoubleCheckedLocking"/> -->
|
||||
<module name="EmptyStatement"/>
|
||||
<module name="EqualsHashCode"/>
|
||||
<!-- <module name="HiddenField"/> -->
|
||||
|
||||
<module name="IllegalInstantiation"/>
|
||||
<module name="InnerAssignment"/>
|
||||
<module name="MagicNumber">
|
||||
<property name="ignoreNumbers" value="-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 1000"/>
|
||||
</module>
|
||||
<module name="MissingSwitchDefault"/>
|
||||
<module name="RedundantThrows"/>
|
||||
<module name="SimplifyBooleanExpression"/>
|
||||
<module name="SimplifyBooleanReturn"/>
|
||||
|
||||
<!-- Checks for class design -->
|
||||
<!-- See http://checkstyle.sf.net/config_design.html -->
|
||||
|
||||
<!-- <module name="DesignForExtension"/> -->
|
||||
|
||||
<module name="FinalClass"/>
|
||||
<module name="HideUtilityClassConstructor"/>
|
||||
<module name="InterfaceIsType"/>
|
||||
<module name="VisibilityModifier"/>
|
||||
|
||||
|
||||
<!-- Miscellaneous other checks. -->
|
||||
<!-- See http://checkstyle.sf.net/config_misc.html -->
|
||||
<module name="ArrayTypeStyle"/>
|
||||
|
||||
<!-- <module name="FinalParameters"/> -->
|
||||
|
||||
<module name="TodoComment"/>
|
||||
<module name="UpperEll"/>
|
||||
|
||||
</module>
|
||||
|
||||
</module>
|
||||
@ -0,0 +1,44 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.classification.InterfaceStability;
|
||||
|
||||
/**
|
||||
* Thrown if there is a problem communicating with Azure Storage service.
|
||||
*/
|
||||
@InterfaceAudience.Public
|
||||
@InterfaceStability.Evolving
|
||||
public class AzureException extends IOException {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
public AzureException(String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
public AzureException(String message, Throwable cause) {
|
||||
super(message, cause);
|
||||
}
|
||||
|
||||
public AzureException(Throwable t) {
|
||||
super(t);
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,42 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
/**
|
||||
* Indicates whether there are actual blobs indicating the existence of
|
||||
* directories or whether we're inferring their existence from them having files
|
||||
* in there.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
enum BlobMaterialization {
|
||||
/**
|
||||
* Indicates a directory that isn't backed by an actual blob, but its
|
||||
* existence is implied by the fact that there are files in there. For
|
||||
* example, if the blob /a/b exists then it implies the existence of the /a
|
||||
* directory if there's no /a blob indicating it.
|
||||
*/
|
||||
Implicit,
|
||||
/**
|
||||
* Indicates that the directory is backed by an actual blob that has the
|
||||
* isFolder metadata on it.
|
||||
*/
|
||||
Explicit,
|
||||
}
|
||||
@ -0,0 +1,120 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.fs.permission.PermissionStatus;
|
||||
|
||||
/**
|
||||
* <p>
|
||||
* Holds basic metadata for a file stored in a {@link NativeFileSystemStore}.
|
||||
* </p>
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
class FileMetadata {
|
||||
private final String key;
|
||||
private final long length;
|
||||
private final long lastModified;
|
||||
private final boolean isDir;
|
||||
private final PermissionStatus permissionStatus;
|
||||
private final BlobMaterialization blobMaterialization;
|
||||
|
||||
/**
|
||||
* Constructs a FileMetadata object for a file.
|
||||
*
|
||||
* @param key
|
||||
* The key (path) to the file.
|
||||
* @param length
|
||||
* The length in bytes of the file.
|
||||
* @param lastModified
|
||||
* The last modified date (milliseconds since January 1, 1970 UTC.)
|
||||
* @param permissionStatus
|
||||
* The permission for the file.
|
||||
*/
|
||||
public FileMetadata(String key, long length, long lastModified,
|
||||
PermissionStatus permissionStatus) {
|
||||
this.key = key;
|
||||
this.length = length;
|
||||
this.lastModified = lastModified;
|
||||
this.isDir = false;
|
||||
this.permissionStatus = permissionStatus;
|
||||
this.blobMaterialization = BlobMaterialization.Explicit; // File are never
|
||||
// implicit.
|
||||
}
|
||||
|
||||
/**
|
||||
* Constructs a FileMetadata object for a directory.
|
||||
*
|
||||
* @param key
|
||||
* The key (path) to the directory.
|
||||
* @param lastModified
|
||||
* The last modified date (milliseconds since January 1, 1970 UTC.)
|
||||
* @param permissionStatus
|
||||
* The permission for the directory.
|
||||
* @param blobMaterialization
|
||||
* Whether this is an implicit (no real blob backing it) or explicit
|
||||
* directory.
|
||||
*/
|
||||
public FileMetadata(String key, long lastModified,
|
||||
PermissionStatus permissionStatus, BlobMaterialization blobMaterialization) {
|
||||
this.key = key;
|
||||
this.isDir = true;
|
||||
this.length = 0;
|
||||
this.lastModified = lastModified;
|
||||
this.permissionStatus = permissionStatus;
|
||||
this.blobMaterialization = blobMaterialization;
|
||||
}
|
||||
|
||||
public boolean isDir() {
|
||||
return isDir;
|
||||
}
|
||||
|
||||
public String getKey() {
|
||||
return key;
|
||||
}
|
||||
|
||||
public long getLength() {
|
||||
return length;
|
||||
}
|
||||
|
||||
public long getLastModified() {
|
||||
return lastModified;
|
||||
}
|
||||
|
||||
public PermissionStatus getPermissionStatus() {
|
||||
return permissionStatus;
|
||||
}
|
||||
|
||||
/**
|
||||
* Indicates whether this is an implicit directory (no real blob backing it)
|
||||
* or an explicit one.
|
||||
*
|
||||
* @return Implicit if this is an implicit directory, or Explicit if it's an
|
||||
* explicit directory or a file.
|
||||
*/
|
||||
public BlobMaterialization getBlobMaterialization() {
|
||||
return blobMaterialization;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "FileMetadata[" + key + ", " + length + ", " + lastModified + ", "
|
||||
+ permissionStatus + "]";
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,43 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
|
||||
/**
|
||||
* The interface that every Azure file system key provider must implement.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public interface KeyProvider {
|
||||
/**
|
||||
* Key providers must implement this method. Given a list of configuration
|
||||
* parameters for the specified Azure storage account, retrieve the plaintext
|
||||
* storage account key.
|
||||
*
|
||||
* @param accountName
|
||||
* the storage account name
|
||||
* @param conf
|
||||
* Hadoop configuration parameters
|
||||
* @return the plaintext storage account key
|
||||
* @throws KeyProviderException
|
||||
*/
|
||||
String getStorageAccountKey(String accountName, Configuration conf)
|
||||
throws KeyProviderException;
|
||||
}
|
||||
@ -0,0 +1,42 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
/**
|
||||
* Thrown if there is a problem instantiating a KeyProvider or retrieving a key
|
||||
* using a KeyProvider object.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public class KeyProviderException extends Exception {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
public KeyProviderException(String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
public KeyProviderException(String message, Throwable cause) {
|
||||
super(message, cause);
|
||||
}
|
||||
|
||||
public KeyProviderException(Throwable t) {
|
||||
super(t);
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,97 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.DataInputStream;
|
||||
import java.io.DataOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.util.Date;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.permission.PermissionStatus;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
/**
|
||||
* <p>
|
||||
* An abstraction for a key-based {@link File} store.
|
||||
* </p>
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
interface NativeFileSystemStore {
|
||||
|
||||
void initialize(URI uri, Configuration conf) throws IOException;
|
||||
|
||||
void storeEmptyFolder(String key, PermissionStatus permissionStatus)
|
||||
throws AzureException;
|
||||
|
||||
FileMetadata retrieveMetadata(String key) throws IOException;
|
||||
|
||||
DataInputStream retrieve(String key) throws IOException;
|
||||
|
||||
DataInputStream retrieve(String key, long byteRangeStart) throws IOException;
|
||||
|
||||
DataOutputStream storefile(String key, PermissionStatus permissionStatus)
|
||||
throws AzureException;
|
||||
|
||||
void storeEmptyLinkFile(String key, String tempBlobKey,
|
||||
PermissionStatus permissionStatus) throws AzureException;
|
||||
|
||||
String getLinkInFileMetadata(String key) throws AzureException;
|
||||
|
||||
PartialListing list(String prefix, final int maxListingCount,
|
||||
final int maxListingDepth) throws IOException;
|
||||
|
||||
PartialListing list(String prefix, final int maxListingCount,
|
||||
final int maxListingDepth, String priorLastKey) throws IOException;
|
||||
|
||||
PartialListing listAll(String prefix, final int maxListingCount,
|
||||
final int maxListingDepth, String priorLastKey) throws IOException;
|
||||
|
||||
void changePermissionStatus(String key, PermissionStatus newPermission)
|
||||
throws AzureException;
|
||||
|
||||
void delete(String key) throws IOException;
|
||||
|
||||
void rename(String srcKey, String dstKey) throws IOException;
|
||||
|
||||
/**
|
||||
* Delete all keys with the given prefix. Used for testing.
|
||||
*
|
||||
* @throws IOException
|
||||
*/
|
||||
@VisibleForTesting
|
||||
void purge(String prefix) throws IOException;
|
||||
|
||||
/**
|
||||
* Diagnostic method to dump state to the console.
|
||||
*
|
||||
* @throws IOException
|
||||
*/
|
||||
void dump() throws IOException;
|
||||
|
||||
void close();
|
||||
|
||||
void updateFolderLastModifiedTime(String key) throws AzureException;
|
||||
|
||||
void updateFolderLastModifiedTime(String key, Date lastModified)
|
||||
throws AzureException;
|
||||
}
|
||||
@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
/**
|
||||
* <p>
|
||||
* Holds information on a directory listing for a {@link NativeFileSystemStore}.
|
||||
* This includes the {@link FileMetadata files} and directories (their names)
|
||||
* contained in a directory.
|
||||
* </p>
|
||||
* <p>
|
||||
* This listing may be returned in chunks, so a <code>priorLastKey</code> is
|
||||
* provided so that the next chunk may be requested.
|
||||
* </p>
|
||||
*
|
||||
* @see NativeFileSystemStore#list(String, int, String)
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
class PartialListing {
|
||||
|
||||
private final String priorLastKey;
|
||||
private final FileMetadata[] files;
|
||||
private final String[] commonPrefixes;
|
||||
|
||||
public PartialListing(String priorLastKey, FileMetadata[] files,
|
||||
String[] commonPrefixes) {
|
||||
this.priorLastKey = priorLastKey;
|
||||
this.files = files;
|
||||
this.commonPrefixes = commonPrefixes;
|
||||
}
|
||||
|
||||
public FileMetadata[] getFiles() {
|
||||
return files;
|
||||
}
|
||||
|
||||
public String[] getCommonPrefixes() {
|
||||
return commonPrefixes;
|
||||
}
|
||||
|
||||
public String getPriorLastKey() {
|
||||
return priorLastKey;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,194 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.net.HttpURLConnection;
|
||||
import java.util.Date;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.RequestResult;
|
||||
import com.microsoft.windowsazure.storage.ResponseReceivedEvent;
|
||||
import com.microsoft.windowsazure.storage.SendingRequestEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageEvent;
|
||||
|
||||
/*
|
||||
* Self throttling is implemented by hooking into send & response callbacks
|
||||
* One instance of this class is created per operationContext so each blobUpload/blobDownload/etc.
|
||||
*
|
||||
* Self throttling only applies to 2nd and subsequent packets of an operation. This is a simple way to
|
||||
* ensure it only affects bulk transfers and not every tiny request.
|
||||
*
|
||||
* A blobDownload will involve sequential packet transmissions and so there are no concurrency concerns
|
||||
* A blobUpload will generally involve concurrent upload worker threads that share one operationContext and one throttling instance.
|
||||
* -- we do not track the latencies for each worker thread as they are doing similar work and will rarely collide in practice.
|
||||
* -- concurrent access to lastE2Edelay must be protected.
|
||||
* -- volatile is necessary and should be sufficient to protect simple access to primitive values (java 1.5 onwards)
|
||||
* -- synchronized{} blocks are also used to be conservative and for easier maintenance.
|
||||
*
|
||||
* If an operation were to perform concurrent GETs and PUTs there is the possibility of getting confused regarding
|
||||
* whether lastE2Edelay was a read or write measurement. This scenario does not occur.
|
||||
*
|
||||
* readFactor = target read throughput as factor of unrestricted throughput.
|
||||
* writeFactor = target write throughput as factor of unrestricted throughput.
|
||||
*
|
||||
* As we introduce delays it is important to only measure the actual E2E latency and not the augmented latency
|
||||
* To achieve this, we fiddle the 'startDate' of the transfer tracking object.
|
||||
*/
|
||||
|
||||
|
||||
/**
|
||||
*
|
||||
* Introduces delays in our Azure traffic to prevent overrunning the server-side throttling limits.
|
||||
*
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public class SelfThrottlingIntercept {
|
||||
public static final Log LOG = LogFactory
|
||||
.getLog(SelfThrottlingIntercept.class);
|
||||
|
||||
private final float readFactor;
|
||||
private final float writeFactor;
|
||||
|
||||
// Concurrency: access to non-final members must be thread-safe
|
||||
private long lastE2Elatency;
|
||||
|
||||
public SelfThrottlingIntercept(OperationContext operationContext,
|
||||
float readFactor, float writeFactor) {
|
||||
this.readFactor = readFactor;
|
||||
this.writeFactor = writeFactor;
|
||||
}
|
||||
|
||||
public static void hook(OperationContext operationContext, float readFactor,
|
||||
float writeFactor) {
|
||||
|
||||
SelfThrottlingIntercept throttler = new SelfThrottlingIntercept(
|
||||
operationContext, readFactor, writeFactor);
|
||||
ResponseReceivedListener responseListener = throttler.new ResponseReceivedListener();
|
||||
SendingRequestListener sendingListener = throttler.new SendingRequestListener();
|
||||
|
||||
operationContext.getResponseReceivedEventHandler().addListener(
|
||||
responseListener);
|
||||
operationContext.getSendingRequestEventHandler().addListener(
|
||||
sendingListener);
|
||||
}
|
||||
|
||||
public void responseReceived(ResponseReceivedEvent event) {
|
||||
RequestResult result = event.getRequestResult();
|
||||
Date startDate = result.getStartDate();
|
||||
Date stopDate = result.getStopDate();
|
||||
long elapsed = stopDate.getTime() - startDate.getTime();
|
||||
|
||||
synchronized (this) {
|
||||
this.lastE2Elatency = elapsed;
|
||||
}
|
||||
|
||||
if (LOG.isDebugEnabled()) {
|
||||
int statusCode = result.getStatusCode();
|
||||
String etag = result.getEtag();
|
||||
HttpURLConnection urlConnection = (HttpURLConnection) event
|
||||
.getConnectionObject();
|
||||
int contentLength = urlConnection.getContentLength();
|
||||
String requestMethod = urlConnection.getRequestMethod();
|
||||
long threadId = Thread.currentThread().getId();
|
||||
LOG.debug(String
|
||||
.format(
|
||||
"SelfThrottlingIntercept:: ResponseReceived: threadId=%d, Status=%d, Elapsed(ms)=%d, ETAG=%s, contentLength=%d, requestMethod=%s",
|
||||
threadId, statusCode, elapsed, etag, contentLength, requestMethod));
|
||||
}
|
||||
}
|
||||
|
||||
public void sendingRequest(SendingRequestEvent sendEvent) {
|
||||
long lastLatency;
|
||||
boolean operationIsRead; // for logging
|
||||
synchronized (this) {
|
||||
|
||||
lastLatency = this.lastE2Elatency;
|
||||
}
|
||||
|
||||
float sleepMultiple;
|
||||
HttpURLConnection urlConnection = (HttpURLConnection) sendEvent
|
||||
.getConnectionObject();
|
||||
|
||||
// Azure REST API never uses POST, so PUT is a sufficient test for an
|
||||
// upload.
|
||||
if (urlConnection.getRequestMethod().equalsIgnoreCase("PUT")) {
|
||||
operationIsRead = false;
|
||||
sleepMultiple = (1 / writeFactor) - 1;
|
||||
} else {
|
||||
operationIsRead = true;
|
||||
sleepMultiple = (1 / readFactor) - 1;
|
||||
}
|
||||
|
||||
long sleepDuration = (long) (sleepMultiple * lastLatency);
|
||||
if (sleepDuration < 0) {
|
||||
sleepDuration = 0;
|
||||
}
|
||||
|
||||
if (sleepDuration > 0) {
|
||||
try {
|
||||
// Thread.sleep() is not exact but it seems sufficiently accurate for
|
||||
// our needs. If needed this could become a loop of small waits that
|
||||
// tracks actual
|
||||
// elapsed time.
|
||||
Thread.sleep(sleepDuration);
|
||||
} catch (InterruptedException ie) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
|
||||
// reset to avoid counting the sleep against request latency
|
||||
sendEvent.getRequestResult().setStartDate(new Date());
|
||||
}
|
||||
|
||||
if (LOG.isDebugEnabled()) {
|
||||
boolean isFirstRequest = (lastLatency == 0);
|
||||
long threadId = Thread.currentThread().getId();
|
||||
LOG.debug(String
|
||||
.format(
|
||||
" SelfThrottlingIntercept:: SendingRequest: threadId=%d, requestType=%s, isFirstRequest=%b, sleepDuration=%d",
|
||||
threadId, operationIsRead ? "read " : "write", isFirstRequest,
|
||||
sleepDuration));
|
||||
}
|
||||
}
|
||||
|
||||
// simply forwards back to the main class.
|
||||
// this is necessary as our main class cannot implement two base-classes.
|
||||
@InterfaceAudience.Private
|
||||
class SendingRequestListener extends StorageEvent<SendingRequestEvent> {
|
||||
|
||||
@Override
|
||||
public void eventOccurred(SendingRequestEvent event) {
|
||||
sendingRequest(event);
|
||||
}
|
||||
}
|
||||
|
||||
// simply forwards back to the main class.
|
||||
// this is necessary as our main class cannot implement two base-classes.
|
||||
@InterfaceAudience.Private
|
||||
class ResponseReceivedListener extends StorageEvent<ResponseReceivedEvent> {
|
||||
|
||||
@Override
|
||||
public void eventOccurred(ResponseReceivedEvent event) {
|
||||
responseReceived(event);
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,166 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.net.HttpURLConnection;
|
||||
import java.security.InvalidKeyException;
|
||||
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
import com.microsoft.windowsazure.storage.Constants.HeaderConstants;
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.SendingRequestEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentials;
|
||||
import com.microsoft.windowsazure.storage.StorageEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageException;
|
||||
|
||||
/**
|
||||
* Manages the lifetime of binding on the operation contexts to intercept send
|
||||
* request events to Azure storage.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public final class SendRequestIntercept extends StorageEvent<SendingRequestEvent> {
|
||||
|
||||
public static final Log LOG = LogFactory.getLog(SendRequestIntercept.class);
|
||||
|
||||
private static final String ALLOW_ALL_REQUEST_PRECONDITIONS = "*";
|
||||
private final StorageCredentials storageCreds;
|
||||
private final boolean allowConcurrentOOBIo;
|
||||
private final OperationContext opContext;
|
||||
|
||||
/**
|
||||
* Getter returning the storage account credentials.
|
||||
*
|
||||
* @return storageCreds - account storage credentials.
|
||||
*/
|
||||
private StorageCredentials getCredentials() {
|
||||
return storageCreds;
|
||||
}
|
||||
|
||||
/**
|
||||
* Query if out-of-band I/Os are allowed.
|
||||
*
|
||||
* return allowConcurrentOOBIo - true if OOB I/O is allowed, and false
|
||||
* otherwise.
|
||||
*/
|
||||
private boolean isOutOfBandIoAllowed() {
|
||||
return allowConcurrentOOBIo;
|
||||
}
|
||||
|
||||
/**
|
||||
* Getter returning the operation context.
|
||||
*
|
||||
* @return storageCreds - account storage credentials.
|
||||
*/
|
||||
private OperationContext getOperationContext() {
|
||||
return opContext;
|
||||
}
|
||||
|
||||
/**
|
||||
* Constructor for SendRequestThrottle.
|
||||
*
|
||||
* @param storageCreds
|
||||
* - storage account credentials for signing packets.
|
||||
*
|
||||
*/
|
||||
private SendRequestIntercept(StorageCredentials storageCreds,
|
||||
boolean allowConcurrentOOBIo, OperationContext opContext) {
|
||||
// Capture the send delay callback interface.
|
||||
this.storageCreds = storageCreds;
|
||||
this.allowConcurrentOOBIo = allowConcurrentOOBIo;
|
||||
this.opContext = opContext;
|
||||
}
|
||||
|
||||
/**
|
||||
* Binds a new lister to the operation context so the WASB file system can
|
||||
* appropriately intercept sends. By allowing concurrent OOB I/Os, we bypass
|
||||
* the blob immutability check when reading streams.
|
||||
*
|
||||
* @param opContext
|
||||
* The operation context to bind to listener.
|
||||
*
|
||||
* @param allowConcurrentOOBIo
|
||||
* True if reads are allowed with concurrent OOB writes.
|
||||
*/
|
||||
public static void bind(StorageCredentials storageCreds,
|
||||
OperationContext opContext, boolean allowConcurrentOOBIo) {
|
||||
SendRequestIntercept sendListener = new SendRequestIntercept(storageCreds,
|
||||
allowConcurrentOOBIo, opContext);
|
||||
opContext.getSendingRequestEventHandler().addListener(sendListener);
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler which processes the sending request event from Azure SDK. The
|
||||
* handler simply sets reset the conditional header to make all read requests
|
||||
* unconditional if reads with concurrent OOB writes are allowed.
|
||||
*
|
||||
* @param sendEvent
|
||||
* - send event context from Windows Azure SDK.
|
||||
*/
|
||||
@Override
|
||||
public void eventOccurred(SendingRequestEvent sendEvent) {
|
||||
|
||||
if (!(sendEvent.getConnectionObject() instanceof HttpURLConnection)) {
|
||||
// Pass if there is no HTTP connection associated with this send
|
||||
// request.
|
||||
return;
|
||||
}
|
||||
|
||||
// Capture the HTTP URL connection object and get size of the payload for
|
||||
// the request.
|
||||
HttpURLConnection urlConnection = (HttpURLConnection) sendEvent
|
||||
.getConnectionObject();
|
||||
|
||||
// Determine whether this is a download request by checking that the request
|
||||
// method
|
||||
// is a "GET" operation.
|
||||
if (urlConnection.getRequestMethod().equalsIgnoreCase("GET")
|
||||
&& isOutOfBandIoAllowed()) {
|
||||
// If concurrent reads on OOB writes are allowed, reset the if-match
|
||||
// condition on the conditional header.
|
||||
urlConnection.setRequestProperty(HeaderConstants.IF_MATCH,
|
||||
ALLOW_ALL_REQUEST_PRECONDITIONS);
|
||||
|
||||
// In the Java AzureSDK the packet is signed before firing the
|
||||
// SendRequest. Setting
|
||||
// the conditional packet header property changes the contents of the
|
||||
// packet, therefore the packet has to be re-signed.
|
||||
try {
|
||||
// Sign the request. GET's have no payload so the content length is
|
||||
// zero.
|
||||
getCredentials().signBlobAndQueueRequest(urlConnection, -1L, getOperationContext());
|
||||
} catch (InvalidKeyException e) {
|
||||
// Log invalid key exception to track signing error before the send
|
||||
// fails.
|
||||
String errString = String.format(
|
||||
"Received invalid key exception when attempting sign packet."
|
||||
+ " Cause: %s", e.getCause().toString());
|
||||
LOG.error(errString);
|
||||
} catch (StorageException e) {
|
||||
// Log storage exception to track signing error before the call fails.
|
||||
String errString = String.format(
|
||||
"Received storage exception when attempting to sign packet."
|
||||
+ " Cause: %s", e.getCause().toString());
|
||||
LOG.error(errString);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.util.Shell;
|
||||
|
||||
/**
|
||||
* Shell decryption key provider which invokes an external script that will
|
||||
* perform the key decryption.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public class ShellDecryptionKeyProvider extends SimpleKeyProvider {
|
||||
static final String KEY_ACCOUNT_SHELLKEYPROVIDER_SCRIPT = "fs.azure.shellkeyprovider.script";
|
||||
|
||||
@Override
|
||||
public String getStorageAccountKey(String accountName, Configuration conf)
|
||||
throws KeyProviderException {
|
||||
String envelope = super.getStorageAccountKey(accountName, conf);
|
||||
|
||||
final String command = conf.get(KEY_ACCOUNT_SHELLKEYPROVIDER_SCRIPT);
|
||||
if (command == null) {
|
||||
throw new KeyProviderException(
|
||||
"Script path is not specified via fs.azure.shellkeyprovider.script");
|
||||
}
|
||||
|
||||
String[] cmd = command.split(" ");
|
||||
String[] cmdWithEnvelope = Arrays.copyOf(cmd, cmd.length + 1);
|
||||
cmdWithEnvelope[cmdWithEnvelope.length - 1] = envelope;
|
||||
|
||||
String decryptedKey = null;
|
||||
try {
|
||||
decryptedKey = Shell.execCommand(cmdWithEnvelope);
|
||||
} catch (IOException ex) {
|
||||
throw new KeyProviderException(ex);
|
||||
}
|
||||
|
||||
// trim any whitespace
|
||||
return decryptedKey.trim();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,42 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
|
||||
/**
|
||||
* Key provider that simply returns the storage account key from the
|
||||
* configuration as plaintext.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public class SimpleKeyProvider implements KeyProvider {
|
||||
|
||||
protected static final String KEY_ACCOUNT_KEY_PREFIX = "fs.azure.account.key.";
|
||||
|
||||
@Override
|
||||
public String getStorageAccountKey(String accountName, Configuration conf)
|
||||
throws KeyProviderException {
|
||||
return conf.get(getStorageAccountKeyName(accountName));
|
||||
}
|
||||
|
||||
protected String getStorageAccountKeyName(String accountName) {
|
||||
return KEY_ACCOUNT_KEY_PREFIX + accountName;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,566 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.util.EnumSet;
|
||||
import java.util.HashMap;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
import com.microsoft.windowsazure.storage.CloudStorageAccount;
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.RetryPolicyFactory;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentials;
|
||||
import com.microsoft.windowsazure.storage.StorageException;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobListingDetails;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobProperties;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobRequestOptions;
|
||||
import com.microsoft.windowsazure.storage.blob.CopyState;
|
||||
import com.microsoft.windowsazure.storage.blob.ListBlobItem;
|
||||
|
||||
/**
|
||||
* This is a very thin layer over the methods exposed by the Windows Azure
|
||||
* Storage SDK that we need for WASB implementation. This base class has a real
|
||||
* implementation that just simply redirects to the SDK, and a memory-backed one
|
||||
* that's used for unit tests.
|
||||
*
|
||||
* IMPORTANT: all the methods here must remain very simple redirects since code
|
||||
* written here can't be properly unit tested.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
abstract class StorageInterface {
|
||||
|
||||
/**
|
||||
* Sets the timeout to use when making requests to the storage service.
|
||||
* <p>
|
||||
* The server timeout interval begins at the time that the complete request
|
||||
* has been received by the service, and the server begins processing the
|
||||
* response. If the timeout interval elapses before the response is returned
|
||||
* to the client, the operation times out. The timeout interval resets with
|
||||
* each retry, if the request is retried.
|
||||
*
|
||||
* The default timeout interval for a request made via the service client is
|
||||
* 90 seconds. You can change this value on the service client by setting this
|
||||
* property, so that all subsequent requests made via the service client will
|
||||
* use the new timeout interval. You can also change this value for an
|
||||
* individual request, by setting the
|
||||
* {@link RequestOptions#timeoutIntervalInMs} property.
|
||||
*
|
||||
* If you are downloading a large blob, you should increase the value of the
|
||||
* timeout beyond the default value.
|
||||
*
|
||||
* @param timeoutInMs
|
||||
* The timeout, in milliseconds, to use when making requests to the
|
||||
* storage service.
|
||||
*/
|
||||
public abstract void setTimeoutInMs(int timeoutInMs);
|
||||
|
||||
/**
|
||||
* Sets the RetryPolicyFactory object to use when making service requests.
|
||||
*
|
||||
* @param retryPolicyFactory
|
||||
* the RetryPolicyFactory object to use when making service requests.
|
||||
*/
|
||||
public abstract void setRetryPolicyFactory(
|
||||
final RetryPolicyFactory retryPolicyFactory);
|
||||
|
||||
/**
|
||||
* Creates a new Blob service client.
|
||||
*
|
||||
*/
|
||||
public abstract void createBlobClient(CloudStorageAccount account);
|
||||
|
||||
/**
|
||||
* Creates an instance of the <code>CloudBlobClient</code> class using the
|
||||
* specified Blob service endpoint.
|
||||
*
|
||||
* @param baseUri
|
||||
* A <code>java.net.URI</code> object that represents the Blob
|
||||
* service endpoint used to create the client.
|
||||
*/
|
||||
public abstract void createBlobClient(URI baseUri);
|
||||
|
||||
/**
|
||||
* Creates an instance of the <code>CloudBlobClient</code> class using the
|
||||
* specified Blob service endpoint and account credentials.
|
||||
*
|
||||
* @param baseUri
|
||||
* A <code>java.net.URI</code> object that represents the Blob
|
||||
* service endpoint used to create the client.
|
||||
* @param credentials
|
||||
* A {@link StorageCredentials} object that represents the account
|
||||
* credentials.
|
||||
*/
|
||||
public abstract void createBlobClient(URI baseUri,
|
||||
StorageCredentials credentials);
|
||||
|
||||
/**
|
||||
* Returns the credentials for the Blob service, as configured for the storage
|
||||
* account.
|
||||
*
|
||||
* @return A {@link StorageCredentials} object that represents the credentials
|
||||
* for this storage account.
|
||||
*/
|
||||
public abstract StorageCredentials getCredentials();
|
||||
|
||||
/**
|
||||
* Returns a reference to a {@link CloudBlobContainerWrapper} object that
|
||||
* represents the cloud blob container for the specified address.
|
||||
*
|
||||
* @param name
|
||||
* A <code>String</code> that represents the name of the container.
|
||||
* @return A {@link CloudBlobContainerWrapper} object that represents a
|
||||
* reference to the cloud blob container.
|
||||
*
|
||||
* @throws URISyntaxException
|
||||
* If the resource URI is invalid.
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract CloudBlobContainerWrapper getContainerReference(String name)
|
||||
throws URISyntaxException, StorageException;
|
||||
|
||||
/**
|
||||
* A thin wrapper over the {@link CloudBlobDirectory} class that simply
|
||||
* redirects calls to the real object except in unit tests.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public abstract static class CloudBlobDirectoryWrapper implements
|
||||
ListBlobItem {
|
||||
/**
|
||||
* Returns the URI for this directory.
|
||||
*
|
||||
* @return A <code>java.net.URI</code> object that represents the URI for
|
||||
* this directory.
|
||||
*/
|
||||
public abstract URI getUri();
|
||||
|
||||
/**
|
||||
* Returns an enumerable collection of blob items whose names begin with the
|
||||
* specified prefix, using the specified flat or hierarchical option,
|
||||
* listing details options, request options, and operation context.
|
||||
*
|
||||
* @param prefix
|
||||
* A <code>String</code> that represents the prefix of the blob
|
||||
* name.
|
||||
* @param useFlatBlobListing
|
||||
* <code>true</code> to indicate that the returned list will be
|
||||
* flat; <code>false</code> to indicate that the returned list will
|
||||
* be hierarchical.
|
||||
* @param listingDetails
|
||||
* A <code>java.util.EnumSet</code> object that contains
|
||||
* {@link BlobListingDetails} values that indicate whether
|
||||
* snapshots, metadata, and/or uncommitted blocks are returned.
|
||||
* Committed blocks are always returned.
|
||||
* @param options
|
||||
* A {@link BlobRequestOptions} object that specifies any
|
||||
* additional options for the request. Specifying <code>null</code>
|
||||
* will use the default request options from the associated service
|
||||
* client ( {@link CloudBlobClient}).
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @return An enumerable collection of {@link ListBlobItem} objects that
|
||||
* represent the block items whose names begin with the specified
|
||||
* prefix in this directory.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
* @throws URISyntaxException
|
||||
* If the resource URI is invalid.
|
||||
*/
|
||||
public abstract Iterable<ListBlobItem> listBlobs(String prefix,
|
||||
boolean useFlatBlobListing, EnumSet<BlobListingDetails> listingDetails,
|
||||
BlobRequestOptions options, OperationContext opContext)
|
||||
throws URISyntaxException, StorageException;
|
||||
}
|
||||
|
||||
/**
|
||||
* A thin wrapper over the {@link CloudBlobContainer} class that simply
|
||||
* redirects calls to the real object except in unit tests.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public abstract static class CloudBlobContainerWrapper {
|
||||
/**
|
||||
* Returns the name of the container.
|
||||
*
|
||||
* @return A <code>String</code> that represents the name of the container.
|
||||
*/
|
||||
public abstract String getName();
|
||||
|
||||
/**
|
||||
* Returns a value that indicates whether the container exists, using the
|
||||
* specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @return <code>true</code> if the container exists, otherwise
|
||||
* <code>false</code>.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract boolean exists(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Returns the metadata for the container.
|
||||
*
|
||||
* @return A <code>java.util.HashMap</code> object that represents the
|
||||
* metadata for the container.
|
||||
*/
|
||||
public abstract HashMap<String, String> getMetadata();
|
||||
|
||||
/**
|
||||
* Sets the metadata for the container.
|
||||
*
|
||||
* @param metadata
|
||||
* A <code>java.util.HashMap</code> object that represents the
|
||||
* metadata being assigned to the container.
|
||||
*/
|
||||
public abstract void setMetadata(HashMap<String, String> metadata);
|
||||
|
||||
/**
|
||||
* Downloads the container's attributes, which consist of metadata and
|
||||
* properties, using the specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Uploads the container's metadata using the specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Creates the container using the specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void create(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Returns a wrapper for a CloudBlobDirectory.
|
||||
*
|
||||
* @param relativePath
|
||||
* A <code>String</code> that represents the name of the directory,
|
||||
* relative to the container
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*
|
||||
* @throws URISyntaxException
|
||||
* If URI syntax exception occurred.
|
||||
*/
|
||||
public abstract CloudBlobDirectoryWrapper getDirectoryReference(
|
||||
String relativePath) throws URISyntaxException, StorageException;
|
||||
|
||||
/**
|
||||
* Returns a wrapper for a CloudBlockBlob.
|
||||
*
|
||||
* @param relativePath
|
||||
* A <code>String</code> that represents the name of the blob,
|
||||
* relative to the container
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*
|
||||
* @throws URISyntaxException
|
||||
* If URI syntax exception occurred.
|
||||
*/
|
||||
public abstract CloudBlockBlobWrapper getBlockBlobReference(
|
||||
String relativePath) throws URISyntaxException, StorageException;
|
||||
}
|
||||
|
||||
/**
|
||||
* A thin wrapper over the {@link CloudBlockBlob} class that simply redirects
|
||||
* calls to the real object except in unit tests.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public abstract static class CloudBlockBlobWrapper implements ListBlobItem {
|
||||
/**
|
||||
* Returns the URI for this blob.
|
||||
*
|
||||
* @return A <code>java.net.URI</code> object that represents the URI for
|
||||
* the blob.
|
||||
*/
|
||||
public abstract URI getUri();
|
||||
|
||||
/**
|
||||
* Returns the metadata for the blob.
|
||||
*
|
||||
* @return A <code>java.util.HashMap</code> object that represents the
|
||||
* metadata for the blob.
|
||||
*/
|
||||
public abstract HashMap<String, String> getMetadata();
|
||||
|
||||
/**
|
||||
* Sets the metadata for the blob.
|
||||
*
|
||||
* @param metadata
|
||||
* A <code>java.util.HashMap</code> object that contains the
|
||||
* metadata being assigned to the blob.
|
||||
*/
|
||||
public abstract void setMetadata(HashMap<String, String> metadata);
|
||||
|
||||
/**
|
||||
* Copies an existing blob's contents, properties, and metadata to this
|
||||
* instance of the <code>CloudBlob</code> class, using the specified
|
||||
* operation context.
|
||||
*
|
||||
* @param sourceBlob
|
||||
* A <code>CloudBlob</code> object that represents the source blob
|
||||
* to copy.
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
* @throws URISyntaxException
|
||||
*
|
||||
*/
|
||||
public abstract void startCopyFromBlob(CloudBlockBlobWrapper sourceBlob,
|
||||
OperationContext opContext) throws StorageException, URISyntaxException;
|
||||
|
||||
/**
|
||||
* Returns the blob's copy state.
|
||||
*
|
||||
* @return A {@link CopyState} object that represents the copy state of the
|
||||
* blob.
|
||||
*/
|
||||
public abstract CopyState getCopyState();
|
||||
|
||||
/**
|
||||
* Deletes the blob using the specified operation context.
|
||||
* <p>
|
||||
* A blob that has snapshots cannot be deleted unless the snapshots are also
|
||||
* deleted. If a blob has snapshots, use the
|
||||
* {@link DeleteSnapshotsOption#DELETE_SNAPSHOTS_ONLY} or
|
||||
* {@link DeleteSnapshotsOption#INCLUDE_SNAPSHOTS} value in the
|
||||
* <code>deleteSnapshotsOption</code> parameter to specify how the snapshots
|
||||
* should be handled when the blob is deleted.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void delete(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Checks to see if the blob exists, using the specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @return <code>true</code> if the blob exists, other wise
|
||||
* <code>false</code>.
|
||||
*
|
||||
* @throws StorageException
|
||||
* f a storage service error occurred.
|
||||
*/
|
||||
public abstract boolean exists(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Populates a blob's properties and metadata using the specified operation
|
||||
* context.
|
||||
* <p>
|
||||
* This method populates the blob's system properties and user-defined
|
||||
* metadata. Before reading a blob's properties or metadata, call this
|
||||
* method or its overload to retrieve the latest values for the blob's
|
||||
* properties and metadata from the Windows Azure storage service.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Returns the blob's properties.
|
||||
*
|
||||
* @return A {@link BlobProperties} object that represents the properties of
|
||||
* the blob.
|
||||
*/
|
||||
public abstract BlobProperties getProperties();
|
||||
|
||||
/**
|
||||
* Opens a blob input stream to download the blob using the specified
|
||||
* operation context.
|
||||
* <p>
|
||||
* Use {@link CloudBlobClient#setStreamMinimumReadSizeInBytes} to configure
|
||||
* the read size.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @return An <code>InputStream</code> object that represents the stream to
|
||||
* use for reading from the blob.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract InputStream openInputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException;
|
||||
|
||||
/**
|
||||
* Creates and opens an output stream to write data to the block blob using
|
||||
* the specified operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @return A {@link BlobOutputStream} object used to write data to the blob.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract OutputStream openOutputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException;
|
||||
|
||||
/**
|
||||
* Uploads the source stream data to the blob, using the specified operation
|
||||
* context.
|
||||
*
|
||||
* @param sourceStream
|
||||
* An <code>InputStream</code> object that represents the input
|
||||
* stream to write to the block blob.
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws IOException
|
||||
* If an I/O error occurred.
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void upload(InputStream sourceStream,
|
||||
OperationContext opContext) throws StorageException, IOException;
|
||||
|
||||
/**
|
||||
* Uploads the blob's metadata to the storage service using the specified
|
||||
* lease ID, request options, and operation context.
|
||||
*
|
||||
* @param opContext
|
||||
* An {@link OperationContext} object that represents the context
|
||||
* for the current operation. This object is used to track requests
|
||||
* to the storage service, and to provide additional runtime
|
||||
* information about the operation.
|
||||
*
|
||||
* @throws StorageException
|
||||
* If a storage service error occurred.
|
||||
*/
|
||||
public abstract void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
public abstract void uploadProperties(OperationContext opContext)
|
||||
throws StorageException;
|
||||
|
||||
/**
|
||||
* Sets the minimum read block size to use with this Blob.
|
||||
*
|
||||
* @param minimumReadSizeBytes
|
||||
* The maximum block size, in bytes, for reading from a block blob
|
||||
* while using a {@link BlobInputStream} object, ranging from 512
|
||||
* bytes to 64 MB, inclusive.
|
||||
*/
|
||||
public abstract void setStreamMinimumReadSizeInBytes(
|
||||
int minimumReadSizeBytes);
|
||||
|
||||
/**
|
||||
* Sets the write block size to use with this Blob.
|
||||
*
|
||||
* @param writeBlockSizeBytes
|
||||
* The maximum block size, in bytes, for writing to a block blob
|
||||
* while using a {@link BlobOutputStream} object, ranging from 1 MB
|
||||
* to 4 MB, inclusive.
|
||||
*
|
||||
* @throws IllegalArgumentException
|
||||
* If <code>writeBlockSizeInBytes</code> is less than 1 MB or
|
||||
* greater than 4 MB.
|
||||
*/
|
||||
public abstract void setWriteBlockSizeInBytes(int writeBlockSizeBytes);
|
||||
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,372 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.util.EnumSet;
|
||||
import java.util.HashMap;
|
||||
import java.util.Iterator;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
|
||||
import com.microsoft.windowsazure.storage.AccessCondition;
|
||||
import com.microsoft.windowsazure.storage.CloudStorageAccount;
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.RetryPolicyFactory;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentials;
|
||||
import com.microsoft.windowsazure.storage.StorageException;
|
||||
import com.microsoft.windowsazure.storage.StorageUri;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobListingDetails;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobProperties;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobRequestOptions;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobClient;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobContainer;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobDirectory;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
import com.microsoft.windowsazure.storage.blob.CopyState;
|
||||
import com.microsoft.windowsazure.storage.blob.DeleteSnapshotsOption;
|
||||
import com.microsoft.windowsazure.storage.blob.ListBlobItem;
|
||||
|
||||
/**
|
||||
* A real implementation of the Azure interaction layer that just redirects
|
||||
* calls to the Windows Azure storage SDK.
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
class StorageInterfaceImpl extends StorageInterface {
|
||||
private CloudBlobClient serviceClient;
|
||||
|
||||
@Override
|
||||
public void setRetryPolicyFactory(final RetryPolicyFactory retryPolicyFactory) {
|
||||
serviceClient.setRetryPolicyFactory(retryPolicyFactory);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setTimeoutInMs(int timeoutInMs) {
|
||||
serviceClient.setTimeoutInMs(timeoutInMs);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(CloudStorageAccount account) {
|
||||
serviceClient = account.createCloudBlobClient();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(URI baseUri) {
|
||||
serviceClient = new CloudBlobClient(baseUri);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(URI baseUri, StorageCredentials credentials) {
|
||||
serviceClient = new CloudBlobClient(baseUri, credentials);
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageCredentials getCredentials() {
|
||||
return serviceClient.getCredentials();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainerWrapper getContainerReference(String uri)
|
||||
throws URISyntaxException, StorageException {
|
||||
return new CloudBlobContainerWrapperImpl(
|
||||
serviceClient.getContainerReference(uri));
|
||||
}
|
||||
|
||||
//
|
||||
// WrappingIterator
|
||||
//
|
||||
|
||||
/**
|
||||
* This iterator wraps every ListBlobItem as they come from the listBlobs()
|
||||
* calls to their proper wrapping objects.
|
||||
*/
|
||||
private static class WrappingIterator implements Iterator<ListBlobItem> {
|
||||
private final Iterator<ListBlobItem> present;
|
||||
|
||||
public WrappingIterator(Iterator<ListBlobItem> present) {
|
||||
this.present = present;
|
||||
}
|
||||
|
||||
public static Iterable<ListBlobItem> wrap(
|
||||
final Iterable<ListBlobItem> present) {
|
||||
return new Iterable<ListBlobItem>() {
|
||||
@Override
|
||||
public Iterator<ListBlobItem> iterator() {
|
||||
return new WrappingIterator(present.iterator());
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext() {
|
||||
return present.hasNext();
|
||||
}
|
||||
|
||||
@Override
|
||||
public ListBlobItem next() {
|
||||
ListBlobItem unwrapped = present.next();
|
||||
if (unwrapped instanceof CloudBlobDirectory) {
|
||||
return new CloudBlobDirectoryWrapperImpl((CloudBlobDirectory) unwrapped);
|
||||
} else if (unwrapped instanceof CloudBlockBlob) {
|
||||
return new CloudBlockBlobWrapperImpl((CloudBlockBlob) unwrapped);
|
||||
} else {
|
||||
return unwrapped;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public void remove() {
|
||||
present.remove();
|
||||
}
|
||||
}
|
||||
|
||||
//
|
||||
// CloudBlobDirectoryWrapperImpl
|
||||
//
|
||||
@InterfaceAudience.Private
|
||||
static class CloudBlobDirectoryWrapperImpl extends CloudBlobDirectoryWrapper {
|
||||
private final CloudBlobDirectory directory;
|
||||
|
||||
public CloudBlobDirectoryWrapperImpl(CloudBlobDirectory directory) {
|
||||
this.directory = directory;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URI getUri() {
|
||||
return directory.getUri();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<ListBlobItem> listBlobs(String prefix,
|
||||
boolean useFlatBlobListing, EnumSet<BlobListingDetails> listingDetails,
|
||||
BlobRequestOptions options, OperationContext opContext)
|
||||
throws URISyntaxException, StorageException {
|
||||
return WrappingIterator.wrap(directory.listBlobs(prefix,
|
||||
useFlatBlobListing, listingDetails, options, opContext));
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainer getContainer() throws URISyntaxException,
|
||||
StorageException {
|
||||
return directory.getContainer();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectory getParent() throws URISyntaxException,
|
||||
StorageException {
|
||||
return directory.getParent();
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageUri getStorageUri() {
|
||||
return directory.getStorageUri();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
//
|
||||
// CloudBlobContainerWrapperImpl
|
||||
//
|
||||
@InterfaceAudience.Private
|
||||
static class CloudBlobContainerWrapperImpl extends CloudBlobContainerWrapper {
|
||||
private final CloudBlobContainer container;
|
||||
|
||||
public CloudBlobContainerWrapperImpl(CloudBlobContainer container) {
|
||||
this.container = container;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getName() {
|
||||
return container.getName();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(OperationContext opContext) throws StorageException {
|
||||
return container.exists(AccessCondition.generateEmptyCondition(), null,
|
||||
opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void create(OperationContext opContext) throws StorageException {
|
||||
container.create(null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public HashMap<String, String> getMetadata() {
|
||||
return container.getMetadata();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMetadata(HashMap<String, String> metadata) {
|
||||
container.setMetadata(metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException {
|
||||
container.downloadAttributes(AccessCondition.generateEmptyCondition(),
|
||||
null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException {
|
||||
container.uploadMetadata(AccessCondition.generateEmptyCondition(), null,
|
||||
opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectoryWrapper getDirectoryReference(String relativePath)
|
||||
throws URISyntaxException, StorageException {
|
||||
|
||||
CloudBlobDirectory dir = container.getDirectoryReference(relativePath);
|
||||
return new CloudBlobDirectoryWrapperImpl(dir);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlockBlobWrapper getBlockBlobReference(String relativePath)
|
||||
throws URISyntaxException, StorageException {
|
||||
|
||||
return new CloudBlockBlobWrapperImpl(
|
||||
container.getBlockBlobReference(relativePath));
|
||||
}
|
||||
}
|
||||
|
||||
//
|
||||
// CloudBlockBlobWrapperImpl
|
||||
//
|
||||
@InterfaceAudience.Private
|
||||
static class CloudBlockBlobWrapperImpl extends CloudBlockBlobWrapper {
|
||||
private final CloudBlockBlob blob;
|
||||
|
||||
public URI getUri() {
|
||||
return blob.getUri();
|
||||
}
|
||||
|
||||
public CloudBlockBlobWrapperImpl(CloudBlockBlob blob) {
|
||||
this.blob = blob;
|
||||
}
|
||||
|
||||
@Override
|
||||
public HashMap<String, String> getMetadata() {
|
||||
return blob.getMetadata();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startCopyFromBlob(CloudBlockBlobWrapper sourceBlob,
|
||||
OperationContext opContext) throws StorageException, URISyntaxException {
|
||||
|
||||
blob.startCopyFromBlob(((CloudBlockBlobWrapperImpl) sourceBlob).blob,
|
||||
null, null, null, opContext);
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public void delete(OperationContext opContext) throws StorageException {
|
||||
blob.delete(DeleteSnapshotsOption.NONE, null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(OperationContext opContext) throws StorageException {
|
||||
return blob.exists(null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException {
|
||||
blob.downloadAttributes(null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public BlobProperties getProperties() {
|
||||
return blob.getProperties();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMetadata(HashMap<String, String> metadata) {
|
||||
blob.setMetadata(metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream openInputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException {
|
||||
return blob.openInputStream(null, options, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutputStream openOutputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException {
|
||||
return blob.openOutputStream(null, options, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void upload(InputStream sourceStream, OperationContext opContext)
|
||||
throws StorageException, IOException {
|
||||
blob.upload(sourceStream, 0, null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainer getContainer() throws URISyntaxException,
|
||||
StorageException {
|
||||
return blob.getContainer();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectory getParent() throws URISyntaxException,
|
||||
StorageException {
|
||||
return blob.getParent();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException {
|
||||
blob.uploadMetadata(null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadProperties(OperationContext opContext)
|
||||
throws StorageException {
|
||||
blob.uploadProperties(null, null, opContext);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setStreamMinimumReadSizeInBytes(int minimumReadSizeBytes) {
|
||||
blob.setStreamMinimumReadSizeInBytes(minimumReadSizeBytes);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setWriteBlockSizeInBytes(int writeBlockSizeBytes) {
|
||||
blob.setStreamWriteSizeInBytes(writeBlockSizeBytes);
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageUri getStorageUri() {
|
||||
return blob.getStorageUri();
|
||||
}
|
||||
|
||||
@Override
|
||||
public CopyState getCopyState() {
|
||||
return blob.getCopyState();
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,48 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.classification.InterfaceStability;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.DelegateToFileSystem;
|
||||
|
||||
|
||||
/**
|
||||
* WASB implementation of AbstractFileSystem.
|
||||
* This impl delegates to the old FileSystem
|
||||
*/
|
||||
@InterfaceAudience.Public
|
||||
@InterfaceStability.Evolving
|
||||
public class Wasb extends DelegateToFileSystem {
|
||||
|
||||
Wasb(final URI theUri, final Configuration conf) throws IOException,
|
||||
URISyntaxException {
|
||||
super(theUri, new NativeAzureFileSystem(), conf, "wasb", false);
|
||||
}
|
||||
|
||||
@Override
|
||||
public int getUriDefaultPort() {
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,196 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.classification.InterfaceStability;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.conf.Configured;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.util.Tool;
|
||||
import org.apache.hadoop.util.ToolRunner;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
/**
|
||||
* An fsck tool implementation for WASB that does various admin/cleanup/recovery
|
||||
* tasks on the WASB file system.
|
||||
*/
|
||||
@InterfaceAudience.Public
|
||||
@InterfaceStability.Evolving
|
||||
public class WasbFsck extends Configured implements Tool {
|
||||
private FileSystem mockFileSystemForTesting = null;
|
||||
private static final String LOST_AND_FOUND_PATH = "/lost+found";
|
||||
private boolean pathNameWarning = false;
|
||||
|
||||
public WasbFsck(Configuration conf) {
|
||||
super(conf);
|
||||
}
|
||||
|
||||
/**
|
||||
* For testing purposes, set the file system to use here instead of relying on
|
||||
* getting it from the FileSystem class based on the URI.
|
||||
*
|
||||
* @param fileSystem
|
||||
* The file system to use.
|
||||
*/
|
||||
@VisibleForTesting
|
||||
public void setMockFileSystemForTesting(FileSystem fileSystem) {
|
||||
this.mockFileSystemForTesting = fileSystem;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int run(String[] args) throws Exception {
|
||||
if (doPrintUsage(Arrays.asList(args))) {
|
||||
printUsage();
|
||||
return -1;
|
||||
}
|
||||
Path pathToCheck = null;
|
||||
boolean doRecover = false;
|
||||
boolean doDelete = false;
|
||||
for (String arg : args) {
|
||||
if (!arg.startsWith("-")) {
|
||||
if (pathToCheck != null) {
|
||||
System.err
|
||||
.println("Can't specify multiple paths to check on the command-line");
|
||||
return 1;
|
||||
}
|
||||
pathToCheck = new Path(arg);
|
||||
} else if (arg.equals("-move")) {
|
||||
doRecover = true;
|
||||
} else if (arg.equals("-delete")) {
|
||||
doDelete = true;
|
||||
}
|
||||
}
|
||||
if (doRecover && doDelete) {
|
||||
System.err
|
||||
.println("Conflicting options: can't specify both -move and -delete.");
|
||||
return 1;
|
||||
}
|
||||
if (pathToCheck == null) {
|
||||
pathToCheck = new Path("/"); // Check everything.
|
||||
}
|
||||
FileSystem fs;
|
||||
if (mockFileSystemForTesting == null) {
|
||||
fs = FileSystem.get(pathToCheck.toUri(), getConf());
|
||||
} else {
|
||||
fs = mockFileSystemForTesting;
|
||||
}
|
||||
|
||||
if (!recursiveCheckChildPathName(fs, fs.makeQualified(pathToCheck))) {
|
||||
pathNameWarning = true;
|
||||
}
|
||||
|
||||
if (!(fs instanceof NativeAzureFileSystem)) {
|
||||
System.err
|
||||
.println("Can only check WASB file system. Instead I'm asked to"
|
||||
+ " check: " + fs.getUri());
|
||||
return 2;
|
||||
}
|
||||
NativeAzureFileSystem wasbFs = (NativeAzureFileSystem) fs;
|
||||
if (doRecover) {
|
||||
System.out.println("Recovering files with dangling data under: "
|
||||
+ pathToCheck);
|
||||
wasbFs.recoverFilesWithDanglingTempData(pathToCheck, new Path(
|
||||
LOST_AND_FOUND_PATH));
|
||||
} else if (doDelete) {
|
||||
System.out.println("Deleting temp files with dangling data under: "
|
||||
+ pathToCheck);
|
||||
wasbFs.deleteFilesWithDanglingTempData(pathToCheck);
|
||||
} else {
|
||||
System.out.println("Please specify -move or -delete");
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
public boolean getPathNameWarning() {
|
||||
return pathNameWarning;
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively check if a given path and its child paths have colons in their
|
||||
* names. It returns true if none of them has a colon or this path does not
|
||||
* exist, and false otherwise.
|
||||
*/
|
||||
private boolean recursiveCheckChildPathName(FileSystem fs, Path p)
|
||||
throws IOException {
|
||||
if (p == null) {
|
||||
return true;
|
||||
}
|
||||
if (!fs.exists(p)) {
|
||||
System.out.println("Path " + p + " does not exist!");
|
||||
return true;
|
||||
}
|
||||
|
||||
if (fs.isFile(p)) {
|
||||
if (containsColon(p)) {
|
||||
System.out.println("Warning: file " + p + " has a colon in its name.");
|
||||
return false;
|
||||
} else {
|
||||
return true;
|
||||
}
|
||||
} else {
|
||||
boolean flag;
|
||||
if (containsColon(p)) {
|
||||
System.out.println("Warning: directory " + p
|
||||
+ " has a colon in its name.");
|
||||
flag = false;
|
||||
} else {
|
||||
flag = true;
|
||||
}
|
||||
FileStatus[] listed = fs.listStatus(p);
|
||||
for (FileStatus l : listed) {
|
||||
if (!recursiveCheckChildPathName(fs, l.getPath())) {
|
||||
flag = false;
|
||||
}
|
||||
}
|
||||
return flag;
|
||||
}
|
||||
}
|
||||
|
||||
private boolean containsColon(Path p) {
|
||||
return p.toUri().getPath().toString().contains(":");
|
||||
}
|
||||
|
||||
private static void printUsage() {
|
||||
System.out.println("Usage: WasbFSck [<path>] [-move | -delete]");
|
||||
System.out.println("\t<path>\tstart checking from this path");
|
||||
System.out.println("\t-move\tmove any files whose upload was interrupted"
|
||||
+ " mid-stream to " + LOST_AND_FOUND_PATH);
|
||||
System.out
|
||||
.println("\t-delete\tdelete any files whose upload was interrupted"
|
||||
+ " mid-stream");
|
||||
ToolRunner.printGenericCommandUsage(System.out);
|
||||
}
|
||||
|
||||
private boolean doPrintUsage(List<String> args) {
|
||||
return args.contains("-H");
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws Exception {
|
||||
int res = ToolRunner.run(new WasbFsck(new Configuration()), args);
|
||||
System.exit(res);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
<html>
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
contributor license agreements. See the NOTICE file distributed with
|
||||
this work for additional information regarding copyright ownership.
|
||||
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
(the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
<body>
|
||||
|
||||
<p>
|
||||
A distributed implementation of {@link
|
||||
org.apache.hadoop.fs.FileSystem} for reading and writing files on
|
||||
<a href="http://store.azure.com">Azure Block Storage</a>.
|
||||
This implementation is blob-based and stores files on Azure in their native form for
|
||||
interoperability with other Azure tools.
|
||||
</p>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@ -0,0 +1,726 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
**/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.DEFAULT_STORAGE_EMULATOR_ACCOUNT_NAME;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.util.Calendar;
|
||||
import java.util.Date;
|
||||
import java.util.EnumSet;
|
||||
import java.util.GregorianCalendar;
|
||||
import java.util.TimeZone;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
|
||||
import com.microsoft.windowsazure.storage.AccessCondition;
|
||||
import com.microsoft.windowsazure.storage.CloudStorageAccount;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentials;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentialsAccountAndKey;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentialsAnonymous;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobContainerPermissions;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobContainerPublicAccessType;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobOutputStream;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobClient;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobContainer;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
import com.microsoft.windowsazure.storage.blob.SharedAccessBlobPermissions;
|
||||
import com.microsoft.windowsazure.storage.blob.SharedAccessBlobPolicy;
|
||||
import com.microsoft.windowsazure.storage.core.Base64;
|
||||
|
||||
/**
|
||||
* Helper class to create WASB file systems backed by either a mock in-memory
|
||||
* implementation or a real Azure Storage account. See RunningLiveWasbTests.txt
|
||||
* for instructions on how to connect to a real Azure Storage account.
|
||||
*/
|
||||
public final class AzureBlobStorageTestAccount {
|
||||
|
||||
private static final String ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key.";
|
||||
private static final String SAS_PROPERTY_NAME = "fs.azure.sas.";
|
||||
private static final String TEST_CONFIGURATION_FILE_NAME = "azure-test.xml";
|
||||
private static final String TEST_ACCOUNT_NAME_PROPERTY_NAME = "fs.azure.test.account.name";
|
||||
public static final String MOCK_ACCOUNT_NAME = "mockAccount.blob.core.windows.net";
|
||||
public static final String MOCK_CONTAINER_NAME = "mockContainer";
|
||||
public static final String WASB_AUTHORITY_DELIMITER = "@";
|
||||
public static final String WASB_SCHEME = "wasb";
|
||||
public static final String PATH_DELIMITER = "/";
|
||||
public static final String AZURE_ROOT_CONTAINER = "$root";
|
||||
public static final String MOCK_WASB_URI = "wasb://" + MOCK_CONTAINER_NAME
|
||||
+ WASB_AUTHORITY_DELIMITER + MOCK_ACCOUNT_NAME + "/";
|
||||
private static final String USE_EMULATOR_PROPERTY_NAME = "fs.azure.test.emulator";
|
||||
|
||||
private static final String KEY_DISABLE_THROTTLING = "fs.azure.disable.bandwidth.throttling";
|
||||
private static final String KEY_READ_TOLERATE_CONCURRENT_APPEND = "fs.azure.io.read.tolerate.concurrent.append";
|
||||
|
||||
private CloudStorageAccount account;
|
||||
private CloudBlobContainer container;
|
||||
private CloudBlockBlob blob;
|
||||
private NativeAzureFileSystem fs;
|
||||
private AzureNativeFileSystemStore storage;
|
||||
private MockStorageInterface mockStorage;
|
||||
|
||||
private AzureBlobStorageTestAccount(NativeAzureFileSystem fs,
|
||||
CloudStorageAccount account, CloudBlobContainer container) {
|
||||
this.account = account;
|
||||
this.container = container;
|
||||
this.fs = fs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a test account with an initialized storage reference.
|
||||
*
|
||||
* @param storage
|
||||
* -- store to be accessed by the account
|
||||
* @param account
|
||||
* -- Windows Azure account object
|
||||
* @param container
|
||||
* -- Windows Azure container object
|
||||
*/
|
||||
private AzureBlobStorageTestAccount(AzureNativeFileSystemStore storage,
|
||||
CloudStorageAccount account, CloudBlobContainer container) {
|
||||
this.account = account;
|
||||
this.container = container;
|
||||
this.storage = storage;
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a test account sessions with the default root container.
|
||||
*
|
||||
* @param fs
|
||||
* - file system, namely WASB file system
|
||||
* @param account
|
||||
* - Windows Azure account object
|
||||
* @param blob
|
||||
* - block blob reference
|
||||
*/
|
||||
private AzureBlobStorageTestAccount(NativeAzureFileSystem fs,
|
||||
CloudStorageAccount account, CloudBlockBlob blob) {
|
||||
|
||||
this.account = account;
|
||||
this.blob = blob;
|
||||
this.fs = fs;
|
||||
}
|
||||
|
||||
private AzureBlobStorageTestAccount(NativeAzureFileSystem fs,
|
||||
MockStorageInterface mockStorage) {
|
||||
this.fs = fs;
|
||||
this.mockStorage = mockStorage;
|
||||
}
|
||||
|
||||
public static String getMockContainerUri() {
|
||||
return String.format("http://%s/%s",
|
||||
AzureBlobStorageTestAccount.MOCK_ACCOUNT_NAME,
|
||||
AzureBlobStorageTestAccount.MOCK_CONTAINER_NAME);
|
||||
}
|
||||
|
||||
public static String toMockUri(String path) {
|
||||
return String.format("http://%s/%s/%s",
|
||||
AzureBlobStorageTestAccount.MOCK_ACCOUNT_NAME,
|
||||
AzureBlobStorageTestAccount.MOCK_CONTAINER_NAME, path);
|
||||
}
|
||||
|
||||
public static String toMockUri(Path path) {
|
||||
// Remove the first SEPARATOR
|
||||
return toMockUri(path.toUri().getRawPath().substring(1));
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the blob reference to the given blob key.
|
||||
*
|
||||
* @param blobKey
|
||||
* The blob key (no initial slash).
|
||||
* @return The blob reference.
|
||||
*/
|
||||
public CloudBlockBlob getBlobReference(String blobKey) throws Exception {
|
||||
return container.getBlockBlobReference(String.format(blobKey));
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquires a short lease on the given blob in this test account.
|
||||
*
|
||||
* @param blobKey
|
||||
* The key to the blob (no initial slash).
|
||||
* @return The lease ID.
|
||||
*/
|
||||
public String acquireShortLease(String blobKey) throws Exception {
|
||||
return getBlobReference(blobKey).acquireLease(60, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Releases the lease on the container.
|
||||
*
|
||||
* @param leaseID
|
||||
* The lease ID.
|
||||
*/
|
||||
public void releaseLease(String leaseID, String blobKey) throws Exception {
|
||||
AccessCondition accessCondition = new AccessCondition();
|
||||
accessCondition.setLeaseID(leaseID);
|
||||
getBlobReference(blobKey).releaseLease(accessCondition);
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount createMock() throws Exception {
|
||||
return createMock(new Configuration());
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount createMock(Configuration conf)
|
||||
throws Exception {
|
||||
AzureNativeFileSystemStore store = new AzureNativeFileSystemStore();
|
||||
MockStorageInterface mockStorage = new MockStorageInterface();
|
||||
store.setAzureStorageInteractionLayer(mockStorage);
|
||||
NativeAzureFileSystem fs = new NativeAzureFileSystem(store);
|
||||
addWasbToConfiguration(conf);
|
||||
setMockAccountKey(conf);
|
||||
// register the fs provider.
|
||||
|
||||
fs.initialize(new URI(MOCK_WASB_URI), conf);
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(fs,
|
||||
mockStorage);
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a test account that goes against the storage emulator.
|
||||
*
|
||||
* @return The test account, or null if the emulator isn't setup.
|
||||
*/
|
||||
public static AzureBlobStorageTestAccount createForEmulator()
|
||||
throws Exception {
|
||||
NativeAzureFileSystem fs = null;
|
||||
CloudBlobContainer container = null;
|
||||
Configuration conf = createTestConfiguration();
|
||||
if (!conf.getBoolean(USE_EMULATOR_PROPERTY_NAME, false)) {
|
||||
// Not configured to test against the storage emulator.
|
||||
System.out.println("Skipping emulator Azure test because configuration "
|
||||
+ "doesn't indicate that it's running."
|
||||
+ " Please see README.txt for guidance.");
|
||||
return null;
|
||||
}
|
||||
CloudStorageAccount account = CloudStorageAccount
|
||||
.getDevelopmentStorageAccount();
|
||||
fs = new NativeAzureFileSystem();
|
||||
String containerName = String.format("wasbtests-%s-%tQ",
|
||||
System.getProperty("user.name"), new Date());
|
||||
container = account.createCloudBlobClient().getContainerReference(
|
||||
containerName);
|
||||
container.create();
|
||||
|
||||
// Set account URI and initialize Azure file system.
|
||||
URI accountUri = createAccountUri(DEFAULT_STORAGE_EMULATOR_ACCOUNT_NAME,
|
||||
containerName);
|
||||
fs.initialize(accountUri, conf);
|
||||
|
||||
// Create test account initializing the appropriate member variables.
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(fs,
|
||||
account, container);
|
||||
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount createOutOfBandStore(
|
||||
int uploadBlockSize, int downloadBlockSize) throws Exception {
|
||||
|
||||
CloudBlobContainer container = null;
|
||||
Configuration conf = createTestConfiguration();
|
||||
CloudStorageAccount account = createTestAccount(conf);
|
||||
if (null == account) {
|
||||
return null;
|
||||
}
|
||||
|
||||
String containerName = String.format("wasbtests-%s-%tQ",
|
||||
System.getProperty("user.name"), new Date());
|
||||
|
||||
// Create the container.
|
||||
container = account.createCloudBlobClient().getContainerReference(
|
||||
containerName);
|
||||
container.create();
|
||||
|
||||
String accountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
|
||||
|
||||
// Ensure that custom throttling is disabled and tolerate concurrent
|
||||
// out-of-band appends.
|
||||
conf.setBoolean(KEY_DISABLE_THROTTLING, true);
|
||||
conf.setBoolean(KEY_READ_TOLERATE_CONCURRENT_APPEND, true);
|
||||
|
||||
// Set account URI and initialize Azure file system.
|
||||
URI accountUri = createAccountUri(accountName, containerName);
|
||||
|
||||
// Create a new AzureNativeFileSystemStore object.
|
||||
AzureNativeFileSystemStore testStorage = new AzureNativeFileSystemStore();
|
||||
|
||||
// Initialize the store with the throttling feedback interfaces.
|
||||
testStorage.initialize(accountUri, conf);
|
||||
|
||||
// Create test account initializing the appropriate member variables.
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(
|
||||
testStorage, account, container);
|
||||
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets the mock account key in the given configuration.
|
||||
*
|
||||
* @param conf
|
||||
* The configuration.
|
||||
*/
|
||||
public static void setMockAccountKey(Configuration conf) {
|
||||
setMockAccountKey(conf, MOCK_ACCOUNT_NAME);
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets the mock account key in the given configuration.
|
||||
*
|
||||
* @param conf
|
||||
* The configuration.
|
||||
*/
|
||||
public static void setMockAccountKey(Configuration conf, String accountName) {
|
||||
conf.set(ACCOUNT_KEY_PROPERTY_NAME + accountName,
|
||||
Base64.encode(new byte[] { 1, 2, 3 }));
|
||||
}
|
||||
|
||||
private static URI createAccountUri(String accountName)
|
||||
throws URISyntaxException {
|
||||
return new URI(WASB_SCHEME + ":" + PATH_DELIMITER + PATH_DELIMITER
|
||||
+ accountName);
|
||||
}
|
||||
|
||||
private static URI createAccountUri(String accountName, String containerName)
|
||||
throws URISyntaxException {
|
||||
return new URI(WASB_SCHEME + ":" + PATH_DELIMITER + PATH_DELIMITER
|
||||
+ containerName + WASB_AUTHORITY_DELIMITER + accountName);
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount create() throws Exception {
|
||||
return create("");
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount create(String containerNameSuffix)
|
||||
throws Exception {
|
||||
return create(containerNameSuffix,
|
||||
EnumSet.of(CreateOptions.CreateContainer));
|
||||
}
|
||||
|
||||
// Create a test account which uses throttling.
|
||||
public static AzureBlobStorageTestAccount createThrottled() throws Exception {
|
||||
return create("",
|
||||
EnumSet.of(CreateOptions.useThrottling, CreateOptions.CreateContainer));
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount create(Configuration conf)
|
||||
throws Exception {
|
||||
return create("", EnumSet.of(CreateOptions.CreateContainer), conf);
|
||||
}
|
||||
|
||||
static CloudStorageAccount createStorageAccount(String accountName,
|
||||
Configuration conf, boolean allowAnonymous) throws URISyntaxException,
|
||||
KeyProviderException {
|
||||
String accountKey = AzureNativeFileSystemStore
|
||||
.getAccountKeyFromConfiguration(accountName, conf);
|
||||
StorageCredentials credentials;
|
||||
if (accountKey == null && allowAnonymous) {
|
||||
credentials = StorageCredentialsAnonymous.ANONYMOUS;
|
||||
} else {
|
||||
credentials = new StorageCredentialsAccountAndKey(
|
||||
accountName.split("\\.")[0], accountKey);
|
||||
}
|
||||
if (credentials == null) {
|
||||
return null;
|
||||
} else {
|
||||
return new CloudStorageAccount(credentials);
|
||||
}
|
||||
}
|
||||
|
||||
private static Configuration createTestConfiguration() {
|
||||
return createTestConfiguration(null);
|
||||
}
|
||||
|
||||
protected static Configuration createTestConfiguration(Configuration conf) {
|
||||
if (conf == null) {
|
||||
conf = new Configuration();
|
||||
}
|
||||
|
||||
conf.addResource(TEST_CONFIGURATION_FILE_NAME);
|
||||
return conf;
|
||||
}
|
||||
|
||||
// for programmatic setting of the wasb configuration.
|
||||
// note that tests can also get the
|
||||
public static void addWasbToConfiguration(Configuration conf) {
|
||||
conf.set("fs.wasb.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem");
|
||||
conf.set("fs.wasbs.impl",
|
||||
"org.apache.hadoop.fs.azure.NativeAzureFileSystem");
|
||||
}
|
||||
|
||||
static CloudStorageAccount createTestAccount() throws URISyntaxException,
|
||||
KeyProviderException {
|
||||
return createTestAccount(createTestConfiguration());
|
||||
}
|
||||
|
||||
static CloudStorageAccount createTestAccount(Configuration conf)
|
||||
throws URISyntaxException, KeyProviderException {
|
||||
String testAccountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
|
||||
if (testAccountName == null) {
|
||||
System.out
|
||||
.println("Skipping live Azure test because of missing test account."
|
||||
+ " Please see README.txt for guidance.");
|
||||
return null;
|
||||
}
|
||||
return createStorageAccount(testAccountName, conf, false);
|
||||
}
|
||||
|
||||
public static enum CreateOptions {
|
||||
UseSas, Readonly, CreateContainer, useThrottling
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount create(String containerNameSuffix,
|
||||
EnumSet<CreateOptions> createOptions) throws Exception {
|
||||
return create(containerNameSuffix, createOptions, null);
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount create(String containerNameSuffix,
|
||||
EnumSet<CreateOptions> createOptions, Configuration initialConfiguration)
|
||||
throws Exception {
|
||||
NativeAzureFileSystem fs = null;
|
||||
CloudBlobContainer container = null;
|
||||
Configuration conf = createTestConfiguration(initialConfiguration);
|
||||
CloudStorageAccount account = createTestAccount(conf);
|
||||
if (account == null) {
|
||||
return null;
|
||||
}
|
||||
fs = new NativeAzureFileSystem();
|
||||
String containerName = String.format("wasbtests-%s-%tQ%s",
|
||||
System.getProperty("user.name"), new Date(), containerNameSuffix);
|
||||
container = account.createCloudBlobClient().getContainerReference(
|
||||
containerName);
|
||||
if (createOptions.contains(CreateOptions.CreateContainer)) {
|
||||
container.create();
|
||||
}
|
||||
String accountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
|
||||
if (createOptions.contains(CreateOptions.UseSas)) {
|
||||
String sas = generateSAS(container,
|
||||
createOptions.contains(CreateOptions.Readonly));
|
||||
if (!createOptions.contains(CreateOptions.CreateContainer)) {
|
||||
// The caller doesn't want the container to be pre-created,
|
||||
// so delete it now that we have generated the SAS.
|
||||
container.delete();
|
||||
}
|
||||
// Remove the account key from the configuration to make sure we don't
|
||||
// cheat and use that.
|
||||
conf.set(ACCOUNT_KEY_PROPERTY_NAME + accountName, "");
|
||||
// Set the SAS key.
|
||||
conf.set(SAS_PROPERTY_NAME + containerName + "." + accountName, sas);
|
||||
}
|
||||
|
||||
// Check if throttling is turned on and set throttling parameters
|
||||
// appropriately.
|
||||
if (createOptions.contains(CreateOptions.useThrottling)) {
|
||||
conf.setBoolean(KEY_DISABLE_THROTTLING, false);
|
||||
} else {
|
||||
conf.setBoolean(KEY_DISABLE_THROTTLING, true);
|
||||
}
|
||||
|
||||
// Set account URI and initialize Azure file system.
|
||||
URI accountUri = createAccountUri(accountName, containerName);
|
||||
fs.initialize(accountUri, conf);
|
||||
|
||||
// Create test account initializing the appropriate member variables.
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(fs,
|
||||
account, container);
|
||||
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
private static String generateContainerName() throws Exception {
|
||||
String containerName = String.format("wasbtests-%s-%tQ",
|
||||
System.getProperty("user.name"), new Date());
|
||||
return containerName;
|
||||
}
|
||||
|
||||
private static String generateSAS(CloudBlobContainer container,
|
||||
boolean readonly) throws Exception {
|
||||
|
||||
// Create a container if it does not exist.
|
||||
container.createIfNotExists();
|
||||
|
||||
// Create a new shared access policy.
|
||||
SharedAccessBlobPolicy sasPolicy = new SharedAccessBlobPolicy();
|
||||
|
||||
// Create a UTC Gregorian calendar value.
|
||||
GregorianCalendar calendar = new GregorianCalendar(
|
||||
TimeZone.getTimeZone("UTC"));
|
||||
|
||||
// Specify the current time as the start time for the shared access
|
||||
// signature.
|
||||
//
|
||||
calendar.setTime(new Date());
|
||||
sasPolicy.setSharedAccessStartTime(calendar.getTime());
|
||||
|
||||
// Use the start time delta one hour as the end time for the shared
|
||||
// access signature.
|
||||
calendar.add(Calendar.HOUR, 10);
|
||||
sasPolicy.setSharedAccessExpiryTime(calendar.getTime());
|
||||
|
||||
if (readonly) {
|
||||
// Set READ permissions
|
||||
sasPolicy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ,
|
||||
SharedAccessBlobPermissions.LIST));
|
||||
} else {
|
||||
// Set READ and WRITE permissions.
|
||||
sasPolicy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ,
|
||||
SharedAccessBlobPermissions.WRITE, SharedAccessBlobPermissions.LIST));
|
||||
}
|
||||
|
||||
// Create the container permissions.
|
||||
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
|
||||
|
||||
// Turn public access to the container off.
|
||||
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.OFF);
|
||||
|
||||
container.uploadPermissions(containerPermissions);
|
||||
|
||||
// Create a shared access signature for the container.
|
||||
String sas = container.generateSharedAccessSignature(sasPolicy, null);
|
||||
// HACK: when the just generated SAS is used straight away, we get an
|
||||
// authorization error intermittently. Sleeping for 1.5 seconds fixes that
|
||||
// on my box.
|
||||
Thread.sleep(1500);
|
||||
|
||||
// Return to caller with the shared access signature.
|
||||
return sas;
|
||||
}
|
||||
|
||||
public static void primePublicContainer(CloudBlobClient blobClient,
|
||||
String accountName, String containerName, String blobName, int fileSize)
|
||||
throws Exception {
|
||||
|
||||
// Create a container if it does not exist. The container name
|
||||
// must be lower case.
|
||||
CloudBlobContainer container = blobClient
|
||||
.getContainerReference(containerName);
|
||||
|
||||
container.createIfNotExists();
|
||||
|
||||
// Create a new shared access policy.
|
||||
SharedAccessBlobPolicy sasPolicy = new SharedAccessBlobPolicy();
|
||||
|
||||
// Set READ and WRITE permissions.
|
||||
sasPolicy.setPermissions(EnumSet.of(SharedAccessBlobPermissions.READ,
|
||||
SharedAccessBlobPermissions.WRITE, SharedAccessBlobPermissions.LIST,
|
||||
SharedAccessBlobPermissions.DELETE));
|
||||
|
||||
// Create the container permissions.
|
||||
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
|
||||
|
||||
// Turn public access to the container off.
|
||||
containerPermissions
|
||||
.setPublicAccess(BlobContainerPublicAccessType.CONTAINER);
|
||||
|
||||
// Set the policy using the values set above.
|
||||
containerPermissions.getSharedAccessPolicies().put("testwasbpolicy",
|
||||
sasPolicy);
|
||||
container.uploadPermissions(containerPermissions);
|
||||
|
||||
// Create a blob output stream.
|
||||
CloudBlockBlob blob = container.getBlockBlobReference(blobName);
|
||||
BlobOutputStream outputStream = blob.openOutputStream();
|
||||
|
||||
outputStream.write(new byte[fileSize]);
|
||||
outputStream.close();
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount createAnonymous(
|
||||
final String blobName, final int fileSize) throws Exception {
|
||||
|
||||
NativeAzureFileSystem fs = null;
|
||||
CloudBlobContainer container = null;
|
||||
Configuration conf = createTestConfiguration(), noTestAccountConf = new Configuration();
|
||||
|
||||
// Set up a session with the cloud blob client to generate SAS and check the
|
||||
// existence of a container and capture the container object.
|
||||
CloudStorageAccount account = createTestAccount(conf);
|
||||
if (account == null) {
|
||||
return null;
|
||||
}
|
||||
CloudBlobClient blobClient = account.createCloudBlobClient();
|
||||
|
||||
// Capture the account URL and the account name.
|
||||
String accountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
|
||||
|
||||
// Generate a container name and create a shared access signature string for
|
||||
// it.
|
||||
//
|
||||
String containerName = generateContainerName();
|
||||
|
||||
// Set up public container with the specified blob name.
|
||||
primePublicContainer(blobClient, accountName, containerName, blobName,
|
||||
fileSize);
|
||||
|
||||
// Capture the blob container object. It should exist after generating the
|
||||
// shared access signature.
|
||||
container = blobClient.getContainerReference(containerName);
|
||||
if (null == container || !container.exists()) {
|
||||
final String errMsg = String
|
||||
.format("Container '%s' expected but not found while creating SAS account.");
|
||||
throw new Exception(errMsg);
|
||||
}
|
||||
|
||||
// Set the account URI.
|
||||
URI accountUri = createAccountUri(accountName, containerName);
|
||||
|
||||
// Initialize the Native Azure file system with anonymous credentials.
|
||||
fs = new NativeAzureFileSystem();
|
||||
fs.initialize(accountUri, noTestAccountConf);
|
||||
|
||||
// Create test account initializing the appropriate member variables.
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(fs,
|
||||
account, container);
|
||||
|
||||
// Return to caller with test account.
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
private static CloudBlockBlob primeRootContainer(CloudBlobClient blobClient,
|
||||
String accountName, String blobName, int fileSize) throws Exception {
|
||||
|
||||
// Create a container if it does not exist. The container name
|
||||
// must be lower case.
|
||||
CloudBlobContainer container = blobClient.getContainerReference("https://"
|
||||
+ accountName + "/" + "$root");
|
||||
container.createIfNotExists();
|
||||
|
||||
// Create a blob output stream.
|
||||
CloudBlockBlob blob = container.getBlockBlobReference(blobName);
|
||||
BlobOutputStream outputStream = blob.openOutputStream();
|
||||
|
||||
outputStream.write(new byte[fileSize]);
|
||||
outputStream.close();
|
||||
|
||||
// Return a reference to the block blob object.
|
||||
return blob;
|
||||
}
|
||||
|
||||
public static AzureBlobStorageTestAccount createRoot(final String blobName,
|
||||
final int fileSize) throws Exception {
|
||||
|
||||
NativeAzureFileSystem fs = null;
|
||||
CloudBlobContainer container = null;
|
||||
Configuration conf = createTestConfiguration();
|
||||
|
||||
// Set up a session with the cloud blob client to generate SAS and check the
|
||||
// existence of a container and capture the container object.
|
||||
CloudStorageAccount account = createTestAccount(conf);
|
||||
if (account == null) {
|
||||
return null;
|
||||
}
|
||||
CloudBlobClient blobClient = account.createCloudBlobClient();
|
||||
|
||||
// Capture the account URL and the account name.
|
||||
String accountName = conf.get(TEST_ACCOUNT_NAME_PROPERTY_NAME);
|
||||
|
||||
// Set up public container with the specified blob name.
|
||||
CloudBlockBlob blobRoot = primeRootContainer(blobClient, accountName,
|
||||
blobName, fileSize);
|
||||
|
||||
// Capture the blob container object. It should exist after generating the
|
||||
// shared access signature.
|
||||
container = blobClient.getContainerReference(AZURE_ROOT_CONTAINER);
|
||||
if (null == container || !container.exists()) {
|
||||
final String errMsg = String
|
||||
.format("Container '%s' expected but not found while creating SAS account.");
|
||||
throw new Exception(errMsg);
|
||||
}
|
||||
|
||||
// Set the account URI without a container name.
|
||||
URI accountUri = createAccountUri(accountName);
|
||||
|
||||
// Initialize the Native Azure file system with anonymous credentials.
|
||||
fs = new NativeAzureFileSystem();
|
||||
fs.initialize(accountUri, conf);
|
||||
|
||||
// Create test account initializing the appropriate member variables.
|
||||
// Set the container value to null for the default root container.
|
||||
AzureBlobStorageTestAccount testAcct = new AzureBlobStorageTestAccount(fs,
|
||||
account, blobRoot);
|
||||
|
||||
// Return to caller with test account.
|
||||
return testAcct;
|
||||
}
|
||||
|
||||
public void closeFileSystem() throws Exception {
|
||||
if (fs != null) {
|
||||
fs.close();
|
||||
}
|
||||
}
|
||||
|
||||
public void cleanup() throws Exception {
|
||||
if (fs != null) {
|
||||
fs.close();
|
||||
fs = null;
|
||||
}
|
||||
if (container != null) {
|
||||
container.deleteIfExists();
|
||||
container = null;
|
||||
}
|
||||
if (blob != null) {
|
||||
// The blob member variable is set for blobs under root containers.
|
||||
// Delete blob objects created for root container tests when cleaning
|
||||
// up the test account.
|
||||
blob.delete();
|
||||
blob = null;
|
||||
}
|
||||
}
|
||||
|
||||
public NativeAzureFileSystem getFileSystem() {
|
||||
return fs;
|
||||
}
|
||||
|
||||
public AzureNativeFileSystemStore getStore() {
|
||||
return this.storage;
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the real blob container backing this account if it's not a mock.
|
||||
*
|
||||
* @return A container, or null if it's a mock.
|
||||
*/
|
||||
public CloudBlobContainer getRealContainer() {
|
||||
return container;
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the real blob account backing this account if it's not a mock.
|
||||
*
|
||||
* @return An account, or null if it's a mock.
|
||||
*/
|
||||
public CloudStorageAccount getRealAccount() {
|
||||
return account;
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the mock storage interface if this account is backed by a mock.
|
||||
*
|
||||
* @return The mock storage, or null if it's backed by a real account.
|
||||
*/
|
||||
public MockStorageInterface getMockStorage() {
|
||||
return mockStorage;
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,146 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.OutputStream;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* A simple memory key-value store to help mock the Windows Azure Storage
|
||||
* implementation for unit testing.
|
||||
*/
|
||||
public class InMemoryBlockBlobStore {
|
||||
private final HashMap<String, Entry> blobs = new HashMap<String, Entry>();
|
||||
private HashMap<String, String> containerMetadata;
|
||||
|
||||
public synchronized Iterable<String> getKeys() {
|
||||
return new ArrayList<String>(blobs.keySet());
|
||||
}
|
||||
|
||||
public static class ListBlobEntry {
|
||||
private final String key;
|
||||
private final HashMap<String, String> metadata;
|
||||
private final int contentLength;
|
||||
|
||||
ListBlobEntry(String key, HashMap<String, String> metadata,
|
||||
int contentLength) {
|
||||
this.key = key;
|
||||
this.metadata = metadata;
|
||||
this.contentLength = contentLength;
|
||||
}
|
||||
|
||||
public String getKey() {
|
||||
return key;
|
||||
}
|
||||
|
||||
public HashMap<String, String> getMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
public int getContentLength() {
|
||||
return contentLength;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* List all the blobs whose key starts with the given prefix.
|
||||
*
|
||||
* @param prefix
|
||||
* The prefix to check.
|
||||
* @param includeMetadata
|
||||
* If set, the metadata in the returned listing will be populated;
|
||||
* otherwise it'll be null.
|
||||
* @return The listing.
|
||||
*/
|
||||
public synchronized Iterable<ListBlobEntry> listBlobs(String prefix,
|
||||
boolean includeMetadata) {
|
||||
ArrayList<ListBlobEntry> list = new ArrayList<ListBlobEntry>();
|
||||
for (Map.Entry<String, Entry> entry : blobs.entrySet()) {
|
||||
if (entry.getKey().startsWith(prefix)) {
|
||||
list.add(new ListBlobEntry(entry.getKey(),
|
||||
includeMetadata ? new HashMap<String, String>(
|
||||
entry.getValue().metadata) : null,
|
||||
entry.getValue().content.length));
|
||||
}
|
||||
}
|
||||
return list;
|
||||
}
|
||||
|
||||
public synchronized byte[] getContent(String key) {
|
||||
return blobs.get(key).content;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public synchronized void setContent(String key, byte[] value,
|
||||
HashMap<String, String> metadata) {
|
||||
blobs
|
||||
.put(key, new Entry(value, (HashMap<String, String>) metadata.clone()));
|
||||
}
|
||||
|
||||
public OutputStream upload(final String key,
|
||||
final HashMap<String, String> metadata) {
|
||||
setContent(key, new byte[0], metadata);
|
||||
return new ByteArrayOutputStream() {
|
||||
@Override
|
||||
public void flush() throws IOException {
|
||||
super.flush();
|
||||
setContent(key, toByteArray(), metadata);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
public synchronized void copy(String sourceKey, String destKey) {
|
||||
blobs.put(destKey, blobs.get(sourceKey));
|
||||
}
|
||||
|
||||
public synchronized void delete(String key) {
|
||||
blobs.remove(key);
|
||||
}
|
||||
|
||||
public synchronized boolean exists(String key) {
|
||||
return blobs.containsKey(key);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public synchronized HashMap<String, String> getMetadata(String key) {
|
||||
return (HashMap<String, String>) blobs.get(key).metadata.clone();
|
||||
}
|
||||
|
||||
public synchronized HashMap<String, String> getContainerMetadata() {
|
||||
return containerMetadata;
|
||||
}
|
||||
|
||||
public synchronized void setContainerMetadata(HashMap<String, String> metadata) {
|
||||
containerMetadata = metadata;
|
||||
}
|
||||
|
||||
private static class Entry {
|
||||
private byte[] content;
|
||||
private HashMap<String, String> metadata;
|
||||
|
||||
public Entry(byte[] content, HashMap<String, String> metadata) {
|
||||
this.content = content;
|
||||
this.metadata = metadata;
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,433 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.net.URISyntaxException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Calendar;
|
||||
import java.util.EnumSet;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.TimeZone;
|
||||
|
||||
import org.apache.commons.httpclient.URIException;
|
||||
import org.apache.commons.httpclient.util.URIUtil;
|
||||
import org.apache.commons.io.output.ByteArrayOutputStream;
|
||||
|
||||
import com.microsoft.windowsazure.storage.CloudStorageAccount;
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.RetryPolicyFactory;
|
||||
import com.microsoft.windowsazure.storage.StorageCredentials;
|
||||
import com.microsoft.windowsazure.storage.StorageException;
|
||||
import com.microsoft.windowsazure.storage.StorageUri;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobListingDetails;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobProperties;
|
||||
import com.microsoft.windowsazure.storage.blob.BlobRequestOptions;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobContainer;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobDirectory;
|
||||
import com.microsoft.windowsazure.storage.blob.CopyState;
|
||||
import com.microsoft.windowsazure.storage.blob.ListBlobItem;
|
||||
|
||||
/**
|
||||
* A mock implementation of the Azure Storage interaction layer for unit tests.
|
||||
* Just does in-memory storage.
|
||||
*/
|
||||
public class MockStorageInterface extends StorageInterface {
|
||||
private InMemoryBlockBlobStore backingStore;
|
||||
private final ArrayList<PreExistingContainer> preExistingContainers = new ArrayList<MockStorageInterface.PreExistingContainer>();
|
||||
private String baseUriString;
|
||||
|
||||
public InMemoryBlockBlobStore getBackingStore() {
|
||||
return backingStore;
|
||||
}
|
||||
|
||||
/**
|
||||
* Mocks the situation where a container already exists before WASB comes in,
|
||||
* i.e. the situation where a user creates a container then mounts WASB on the
|
||||
* pre-existing container.
|
||||
*
|
||||
* @param uri
|
||||
* The URI of the container.
|
||||
* @param metadata
|
||||
* The metadata on the container.
|
||||
*/
|
||||
public void addPreExistingContainer(String uri,
|
||||
HashMap<String, String> metadata) {
|
||||
preExistingContainers.add(new PreExistingContainer(uri, metadata));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setRetryPolicyFactory(final RetryPolicyFactory retryPolicyFactory) {
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setTimeoutInMs(int timeoutInMs) {
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(CloudStorageAccount account) {
|
||||
backingStore = new InMemoryBlockBlobStore();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(URI baseUri) {
|
||||
backingStore = new InMemoryBlockBlobStore();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void createBlobClient(URI baseUri, StorageCredentials credentials) {
|
||||
this.baseUriString = baseUri.toString();
|
||||
backingStore = new InMemoryBlockBlobStore();
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageCredentials getCredentials() {
|
||||
// Not implemented for mock interface.
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainerWrapper getContainerReference(String name)
|
||||
throws URISyntaxException, StorageException {
|
||||
String fullUri;
|
||||
try {
|
||||
fullUri = baseUriString + "/" + URIUtil.encodePath(name);
|
||||
} catch (URIException e) {
|
||||
throw new RuntimeException("problem encoding fullUri", e);
|
||||
}
|
||||
|
||||
MockCloudBlobContainerWrapper container = new MockCloudBlobContainerWrapper(
|
||||
fullUri, name);
|
||||
// Check if we have a pre-existing container with that name, and prime
|
||||
// the wrapper with that knowledge if it's found.
|
||||
for (PreExistingContainer existing : preExistingContainers) {
|
||||
if (fullUri.equalsIgnoreCase(existing.containerUri)) {
|
||||
// We have a pre-existing container. Mark the wrapper as created and
|
||||
// make sure we use the metadata for it.
|
||||
container.created = true;
|
||||
backingStore.setContainerMetadata(existing.containerMetadata);
|
||||
break;
|
||||
}
|
||||
}
|
||||
return container;
|
||||
}
|
||||
|
||||
class MockCloudBlobContainerWrapper extends CloudBlobContainerWrapper {
|
||||
private boolean created = false;
|
||||
private HashMap<String, String> metadata;
|
||||
private final String baseUri;
|
||||
private final String name;
|
||||
|
||||
public MockCloudBlobContainerWrapper(String baseUri, String name) {
|
||||
this.baseUri = baseUri;
|
||||
this.name = name;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getName() {
|
||||
return name;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(OperationContext opContext) throws StorageException {
|
||||
return created;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void create(OperationContext opContext) throws StorageException {
|
||||
created = true;
|
||||
backingStore.setContainerMetadata(metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public HashMap<String, String> getMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMetadata(HashMap<String, String> metadata) {
|
||||
this.metadata = metadata;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException {
|
||||
metadata = backingStore.getContainerMetadata();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException {
|
||||
backingStore.setContainerMetadata(metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectoryWrapper getDirectoryReference(String relativePath)
|
||||
throws URISyntaxException, StorageException {
|
||||
return new MockCloudBlobDirectoryWrapper(new URI(fullUriString(
|
||||
relativePath, true)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlockBlobWrapper getBlockBlobReference(String relativePath)
|
||||
throws URISyntaxException, StorageException {
|
||||
return new MockCloudBlockBlobWrapper(new URI(fullUriString(relativePath,
|
||||
false)), null, 0);
|
||||
}
|
||||
|
||||
// helper to create full URIs for directory and blob.
|
||||
// use withTrailingSlash=true to get a good path for a directory.
|
||||
private String fullUriString(String relativePath, boolean withTrailingSlash) {
|
||||
String fullUri;
|
||||
|
||||
String baseUri = this.baseUri;
|
||||
if (!baseUri.endsWith("/")) {
|
||||
baseUri += "/";
|
||||
}
|
||||
if (withTrailingSlash && !relativePath.equals("")
|
||||
&& !relativePath.endsWith("/")) {
|
||||
relativePath += "/";
|
||||
}
|
||||
|
||||
try {
|
||||
fullUri = baseUri + URIUtil.encodePath(relativePath);
|
||||
} catch (URIException e) {
|
||||
throw new RuntimeException("problem encoding fullUri", e);
|
||||
}
|
||||
|
||||
return fullUri;
|
||||
}
|
||||
}
|
||||
|
||||
private static class PreExistingContainer {
|
||||
final String containerUri;
|
||||
final HashMap<String, String> containerMetadata;
|
||||
|
||||
public PreExistingContainer(String uri, HashMap<String, String> metadata) {
|
||||
this.containerUri = uri;
|
||||
this.containerMetadata = metadata;
|
||||
}
|
||||
}
|
||||
|
||||
class MockCloudBlobDirectoryWrapper extends CloudBlobDirectoryWrapper {
|
||||
private URI uri;
|
||||
|
||||
public MockCloudBlobDirectoryWrapper(URI uri) {
|
||||
this.uri = uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainer getContainer() throws URISyntaxException,
|
||||
StorageException {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectory getParent() throws URISyntaxException,
|
||||
StorageException {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URI getUri() {
|
||||
return uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<ListBlobItem> listBlobs(String prefix,
|
||||
boolean useFlatBlobListing, EnumSet<BlobListingDetails> listingDetails,
|
||||
BlobRequestOptions options, OperationContext opContext)
|
||||
throws URISyntaxException, StorageException {
|
||||
ArrayList<ListBlobItem> ret = new ArrayList<ListBlobItem>();
|
||||
String fullPrefix = prefix == null ? uri.toString() : new URI(
|
||||
uri.getScheme(), uri.getAuthority(), uri.getPath() + prefix,
|
||||
uri.getQuery(), uri.getFragment()).toString();
|
||||
boolean includeMetadata = listingDetails
|
||||
.contains(BlobListingDetails.METADATA);
|
||||
HashSet<String> addedDirectories = new HashSet<String>();
|
||||
for (InMemoryBlockBlobStore.ListBlobEntry current : backingStore
|
||||
.listBlobs(fullPrefix, includeMetadata)) {
|
||||
int indexOfSlash = current.getKey().indexOf('/', fullPrefix.length());
|
||||
if (useFlatBlobListing || indexOfSlash < 0) {
|
||||
ret.add(new MockCloudBlockBlobWrapper(new URI(current.getKey()),
|
||||
current.getMetadata(), current.getContentLength()));
|
||||
} else {
|
||||
String directoryName = current.getKey().substring(0, indexOfSlash);
|
||||
if (!addedDirectories.contains(directoryName)) {
|
||||
addedDirectories.add(current.getKey());
|
||||
ret.add(new MockCloudBlobDirectoryWrapper(new URI(directoryName
|
||||
+ "/")));
|
||||
}
|
||||
}
|
||||
}
|
||||
return ret;
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageUri getStorageUri() {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
class MockCloudBlockBlobWrapper extends CloudBlockBlobWrapper {
|
||||
private URI uri;
|
||||
private HashMap<String, String> metadata = new HashMap<String, String>();
|
||||
private BlobProperties properties;
|
||||
|
||||
public MockCloudBlockBlobWrapper(URI uri, HashMap<String, String> metadata,
|
||||
int length) {
|
||||
this.uri = uri;
|
||||
this.metadata = metadata;
|
||||
this.properties = new BlobProperties();
|
||||
this.properties.setLength(length);
|
||||
this.properties.setLastModified(Calendar.getInstance(
|
||||
TimeZone.getTimeZone("UTC")).getTime());
|
||||
}
|
||||
|
||||
private void refreshProperties(boolean getMetadata) {
|
||||
if (backingStore.exists(uri.toString())) {
|
||||
byte[] content = backingStore.getContent(uri.toString());
|
||||
properties = new BlobProperties();
|
||||
properties.setLength(content.length);
|
||||
properties.setLastModified(Calendar.getInstance(
|
||||
TimeZone.getTimeZone("UTC")).getTime());
|
||||
if (getMetadata) {
|
||||
metadata = backingStore.getMetadata(uri.toString());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobContainer getContainer() throws URISyntaxException,
|
||||
StorageException {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloudBlobDirectory getParent() throws URISyntaxException,
|
||||
StorageException {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public URI getUri() {
|
||||
return uri;
|
||||
}
|
||||
|
||||
@Override
|
||||
public HashMap<String, String> getMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setMetadata(HashMap<String, String> metadata) {
|
||||
this.metadata = metadata;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void startCopyFromBlob(CloudBlockBlobWrapper sourceBlob,
|
||||
OperationContext opContext) throws StorageException, URISyntaxException {
|
||||
backingStore.copy(sourceBlob.getUri().toString(), uri.toString());
|
||||
// it would be best if backingStore.properties.CopyState were tracked
|
||||
// If implemented, update azureNativeFileSystemStore.waitForCopyToComplete
|
||||
}
|
||||
|
||||
@Override
|
||||
public CopyState getCopyState() {
|
||||
return this.properties.getCopyState();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void delete(OperationContext opContext) throws StorageException {
|
||||
backingStore.delete(uri.toString());
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(OperationContext opContext) throws StorageException {
|
||||
return backingStore.exists(uri.toString());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void downloadAttributes(OperationContext opContext)
|
||||
throws StorageException {
|
||||
refreshProperties(true);
|
||||
}
|
||||
|
||||
@Override
|
||||
public BlobProperties getProperties() {
|
||||
return properties;
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream openInputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException {
|
||||
return new ByteArrayInputStream(backingStore.getContent(uri.toString()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public OutputStream openOutputStream(BlobRequestOptions options,
|
||||
OperationContext opContext) throws StorageException {
|
||||
return backingStore.upload(uri.toString(), metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void upload(InputStream sourceStream, OperationContext opContext)
|
||||
throws StorageException, IOException {
|
||||
ByteArrayOutputStream allContent = new ByteArrayOutputStream();
|
||||
allContent.write(sourceStream);
|
||||
backingStore.setContent(uri.toString(), allContent.toByteArray(),
|
||||
metadata);
|
||||
refreshProperties(false);
|
||||
allContent.close();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadMetadata(OperationContext opContext)
|
||||
throws StorageException {
|
||||
backingStore.setContent(uri.toString(),
|
||||
backingStore.getContent(uri.toString()), metadata);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void uploadProperties(OperationContext opContext)
|
||||
throws StorageException {
|
||||
refreshProperties(false);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setStreamMinimumReadSizeInBytes(int minimumReadSize) {
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setWriteBlockSizeInBytes(int writeBlockSizeInBytes) {
|
||||
}
|
||||
|
||||
@Override
|
||||
public StorageUri getStorageUri() {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,584 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.BufferedWriter;
|
||||
import java.io.FileNotFoundException;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.io.OutputStream;
|
||||
import java.io.OutputStreamWriter;
|
||||
import java.util.Calendar;
|
||||
import java.util.Date;
|
||||
import java.util.TimeZone;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.FSDataInputStream;
|
||||
import org.apache.hadoop.fs.FSDataOutputStream;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.FileUtil;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.permission.FsPermission;
|
||||
import org.apache.hadoop.security.UserGroupInformation;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
/*
|
||||
* Tests the Native Azure file system (WASB) against an actual blob store if
|
||||
* provided in the environment.
|
||||
* Subclasses implement createTestAccount() to hit local&mock storage with the same test code.
|
||||
*
|
||||
* For hand-testing: remove "abstract" keyword and copy in an implementation of createTestAccount
|
||||
* from one of the subclasses
|
||||
*/
|
||||
public abstract class NativeAzureFileSystemBaseTest {
|
||||
|
||||
private FileSystem fs;
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
private final long modifiedTimeErrorMargin = 5 * 1000; // Give it +/-5 seconds
|
||||
|
||||
protected abstract AzureBlobStorageTestAccount createTestAccount()
|
||||
throws Exception;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = createTestAccount();
|
||||
if (testAccount != null) {
|
||||
fs = testAccount.getFileSystem();
|
||||
}
|
||||
assumeNotNull(testAccount);
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
fs = null;
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCheckingNonExistentOneLetterFile() throws Exception {
|
||||
assertFalse(fs.exists(new Path("/a")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStoreRetrieveFile() throws Exception {
|
||||
Path testFile = new Path("unit-test-file");
|
||||
writeString(testFile, "Testing");
|
||||
assertTrue(fs.exists(testFile));
|
||||
FileStatus status = fs.getFileStatus(testFile);
|
||||
assertNotNull(status);
|
||||
// By default, files should be have masked permissions
|
||||
// that grant RW to user, and R to group/other
|
||||
assertEquals(new FsPermission((short) 0644), status.getPermission());
|
||||
assertEquals("Testing", readString(testFile));
|
||||
fs.delete(testFile, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStoreDeleteFolder() throws Exception {
|
||||
Path testFolder = new Path("storeDeleteFolder");
|
||||
assertFalse(fs.exists(testFolder));
|
||||
assertTrue(fs.mkdirs(testFolder));
|
||||
assertTrue(fs.exists(testFolder));
|
||||
FileStatus status = fs.getFileStatus(testFolder);
|
||||
assertNotNull(status);
|
||||
assertTrue(status.isDirectory());
|
||||
// By default, directories should be have masked permissions
|
||||
// that grant RWX to user, and RX to group/other
|
||||
assertEquals(new FsPermission((short) 0755), status.getPermission());
|
||||
Path innerFile = new Path(testFolder, "innerFile");
|
||||
assertTrue(fs.createNewFile(innerFile));
|
||||
assertTrue(fs.exists(innerFile));
|
||||
assertTrue(fs.delete(testFolder, true));
|
||||
assertFalse(fs.exists(innerFile));
|
||||
assertFalse(fs.exists(testFolder));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFileOwnership() throws Exception {
|
||||
Path testFile = new Path("ownershipTestFile");
|
||||
writeString(testFile, "Testing");
|
||||
testOwnership(testFile);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFolderOwnership() throws Exception {
|
||||
Path testFolder = new Path("ownershipTestFolder");
|
||||
fs.mkdirs(testFolder);
|
||||
testOwnership(testFolder);
|
||||
}
|
||||
|
||||
private void testOwnership(Path pathUnderTest) throws IOException {
|
||||
FileStatus ret = fs.getFileStatus(pathUnderTest);
|
||||
UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
|
||||
assertEquals(ret.getOwner(), currentUser.getShortUserName());
|
||||
fs.delete(pathUnderTest, true);
|
||||
}
|
||||
|
||||
private static FsPermission ignoreStickyBit(FsPermission original) {
|
||||
return new FsPermission(original.getUserAction(),
|
||||
original.getGroupAction(), original.getOtherAction());
|
||||
}
|
||||
|
||||
// When FsPermission applies a UMask, it loses sticky bit information.
|
||||
// And since we always apply UMask, we should ignore whether the sticky
|
||||
// bit is equal or not.
|
||||
private static void assertEqualsIgnoreStickyBit(FsPermission expected,
|
||||
FsPermission actual) {
|
||||
assertEquals(ignoreStickyBit(expected), ignoreStickyBit(actual));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFilePermissions() throws Exception {
|
||||
Path testFile = new Path("permissionTestFile");
|
||||
FsPermission permission = FsPermission.createImmutable((short) 644);
|
||||
createEmptyFile(testFile, permission);
|
||||
FileStatus ret = fs.getFileStatus(testFile);
|
||||
assertEqualsIgnoreStickyBit(permission, ret.getPermission());
|
||||
fs.delete(testFile, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFolderPermissions() throws Exception {
|
||||
Path testFolder = new Path("permissionTestFolder");
|
||||
FsPermission permission = FsPermission.createImmutable((short) 644);
|
||||
fs.mkdirs(testFolder, permission);
|
||||
FileStatus ret = fs.getFileStatus(testFolder);
|
||||
assertEqualsIgnoreStickyBit(permission, ret.getPermission());
|
||||
fs.delete(testFolder, true);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testDeepFileCreation() throws Exception {
|
||||
Path testFile = new Path("deep/file/creation/test");
|
||||
FsPermission permission = FsPermission.createImmutable((short) 644);
|
||||
createEmptyFile(testFile, permission);
|
||||
assertTrue(fs.exists(testFile));
|
||||
assertTrue(fs.exists(new Path("deep")));
|
||||
assertTrue(fs.exists(new Path("deep/file/creation")));
|
||||
FileStatus ret = fs.getFileStatus(new Path("deep/file"));
|
||||
assertTrue(ret.isDirectory());
|
||||
assertEqualsIgnoreStickyBit(permission, ret.getPermission());
|
||||
assertTrue(fs.delete(new Path("deep"), true));
|
||||
assertFalse(fs.exists(testFile));
|
||||
|
||||
// An alternative test scenario would've been to delete the file first,
|
||||
// and then check for the existence of the upper folders still. But that
|
||||
// doesn't actually work as expected right now.
|
||||
}
|
||||
|
||||
private static enum RenameVariation {
|
||||
NormalFileName, SourceInAFolder, SourceWithSpace, SourceWithPlusAndPercent
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testRename() throws Exception {
|
||||
for (RenameVariation variation : RenameVariation.values()) {
|
||||
System.out.printf("Rename variation: %s\n", variation);
|
||||
Path originalFile;
|
||||
switch (variation) {
|
||||
case NormalFileName:
|
||||
originalFile = new Path("fileToRename");
|
||||
break;
|
||||
case SourceInAFolder:
|
||||
originalFile = new Path("file/to/rename");
|
||||
break;
|
||||
case SourceWithSpace:
|
||||
originalFile = new Path("file to rename");
|
||||
break;
|
||||
case SourceWithPlusAndPercent:
|
||||
originalFile = new Path("file+to%rename");
|
||||
break;
|
||||
default:
|
||||
throw new Exception("Unknown variation");
|
||||
}
|
||||
Path destinationFile = new Path("file/resting/destination");
|
||||
assertTrue(fs.createNewFile(originalFile));
|
||||
assertTrue(fs.exists(originalFile));
|
||||
assertFalse(fs.rename(originalFile, destinationFile)); // Parent directory
|
||||
// doesn't exist
|
||||
assertTrue(fs.mkdirs(destinationFile.getParent()));
|
||||
assertTrue(fs.rename(originalFile, destinationFile));
|
||||
assertTrue(fs.exists(destinationFile));
|
||||
assertFalse(fs.exists(originalFile));
|
||||
fs.delete(destinationFile.getParent(), true);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testRenameImplicitFolder() throws Exception {
|
||||
Path testFile = new Path("deep/file/rename/test");
|
||||
FsPermission permission = FsPermission.createImmutable((short) 644);
|
||||
createEmptyFile(testFile, permission);
|
||||
assertTrue(fs.rename(new Path("deep/file"), new Path("deep/renamed")));
|
||||
assertFalse(fs.exists(testFile));
|
||||
FileStatus newStatus = fs
|
||||
.getFileStatus(new Path("deep/renamed/rename/test"));
|
||||
assertNotNull(newStatus);
|
||||
assertEqualsIgnoreStickyBit(permission, newStatus.getPermission());
|
||||
assertTrue(fs.delete(new Path("deep"), true));
|
||||
}
|
||||
|
||||
private static enum RenameFolderVariation {
|
||||
CreateFolderAndInnerFile, CreateJustInnerFile, CreateJustFolder
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testRenameFolder() throws Exception {
|
||||
for (RenameFolderVariation variation : RenameFolderVariation.values()) {
|
||||
Path originalFolder = new Path("folderToRename");
|
||||
if (variation != RenameFolderVariation.CreateJustInnerFile){
|
||||
assertTrue(fs.mkdirs(originalFolder));
|
||||
}
|
||||
Path innerFile = new Path(originalFolder, "innerFile");
|
||||
if (variation != RenameFolderVariation.CreateJustFolder){
|
||||
assertTrue(fs.createNewFile(innerFile));
|
||||
}
|
||||
Path destination = new Path("renamedFolder");
|
||||
assertTrue(fs.rename(originalFolder, destination));
|
||||
assertTrue(fs.exists(destination));
|
||||
if (variation != RenameFolderVariation.CreateJustFolder){
|
||||
assertTrue(fs.exists(new Path(destination, innerFile.getName())));
|
||||
}
|
||||
assertFalse(fs.exists(originalFolder));
|
||||
assertFalse(fs.exists(innerFile));
|
||||
fs.delete(destination, true);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCopyFromLocalFileSystem() throws Exception {
|
||||
Path localFilePath = new Path(System.getProperty("test.build.data",
|
||||
"azure_test"));
|
||||
FileSystem localFs = FileSystem.get(new Configuration());
|
||||
localFs.delete(localFilePath, true);
|
||||
try {
|
||||
writeString(localFs, localFilePath, "Testing");
|
||||
Path dstPath = new Path("copiedFromLocal");
|
||||
assertTrue(FileUtil.copy(localFs, localFilePath, fs, dstPath, false,
|
||||
fs.getConf()));
|
||||
assertTrue(fs.exists(dstPath));
|
||||
assertEquals("Testing", readString(fs, dstPath));
|
||||
fs.delete(dstPath, true);
|
||||
} finally {
|
||||
localFs.delete(localFilePath, true);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testListDirectory() throws Exception {
|
||||
Path rootFolder = new Path("testingList");
|
||||
assertTrue(fs.mkdirs(rootFolder));
|
||||
FileStatus[] listed = fs.listStatus(rootFolder);
|
||||
assertEquals(0, listed.length);
|
||||
Path innerFolder = new Path(rootFolder, "inner");
|
||||
assertTrue(fs.mkdirs(innerFolder));
|
||||
listed = fs.listStatus(rootFolder);
|
||||
assertEquals(1, listed.length);
|
||||
assertTrue(listed[0].isDirectory());
|
||||
Path innerFile = new Path(innerFolder, "innerFile");
|
||||
writeString(innerFile, "testing");
|
||||
listed = fs.listStatus(rootFolder);
|
||||
assertEquals(1, listed.length);
|
||||
assertTrue(listed[0].isDirectory());
|
||||
listed = fs.listStatus(innerFolder);
|
||||
assertEquals(1, listed.length);
|
||||
assertFalse(listed[0].isDirectory());
|
||||
assertTrue(fs.delete(rootFolder, true));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testStatistics() throws Exception {
|
||||
FileSystem.clearStatistics();
|
||||
FileSystem.Statistics stats = FileSystem.getStatistics("wasb",
|
||||
NativeAzureFileSystem.class);
|
||||
assertEquals(0, stats.getBytesRead());
|
||||
assertEquals(0, stats.getBytesWritten());
|
||||
Path newFile = new Path("testStats");
|
||||
writeString(newFile, "12345678");
|
||||
assertEquals(8, stats.getBytesWritten());
|
||||
assertEquals(0, stats.getBytesRead());
|
||||
String readBack = readString(newFile);
|
||||
assertEquals("12345678", readBack);
|
||||
assertEquals(8, stats.getBytesRead());
|
||||
assertEquals(8, stats.getBytesWritten());
|
||||
assertTrue(fs.delete(newFile, true));
|
||||
assertEquals(8, stats.getBytesRead());
|
||||
assertEquals(8, stats.getBytesWritten());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUriEncoding() throws Exception {
|
||||
fs.create(new Path("p/t%5Fe")).close();
|
||||
FileStatus[] listing = fs.listStatus(new Path("p"));
|
||||
assertEquals(1, listing.length);
|
||||
assertEquals("t%5Fe", listing[0].getPath().getName());
|
||||
assertTrue(fs.rename(new Path("p"), new Path("q")));
|
||||
assertTrue(fs.delete(new Path("q"), true));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testUriEncodingMoreComplexCharacters() throws Exception {
|
||||
// Create a file name with URI reserved characters, plus the percent
|
||||
String fileName = "!#$'()*;=[]%";
|
||||
String directoryName = "*;=[]%!#$'()";
|
||||
fs.create(new Path(directoryName, fileName)).close();
|
||||
FileStatus[] listing = fs.listStatus(new Path(directoryName));
|
||||
assertEquals(1, listing.length);
|
||||
assertEquals(fileName, listing[0].getPath().getName());
|
||||
FileStatus status = fs.getFileStatus(new Path(directoryName, fileName));
|
||||
assertEquals(fileName, status.getPath().getName());
|
||||
InputStream stream = fs.open(new Path(directoryName, fileName));
|
||||
assertNotNull(stream);
|
||||
stream.close();
|
||||
assertTrue(fs.delete(new Path(directoryName, fileName), true));
|
||||
assertTrue(fs.delete(new Path(directoryName), true));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testReadingDirectoryAsFile() throws Exception {
|
||||
Path dir = new Path("/x");
|
||||
assertTrue(fs.mkdirs(dir));
|
||||
try {
|
||||
fs.open(dir).close();
|
||||
assertTrue("Should've thrown", false);
|
||||
} catch (FileNotFoundException ex) {
|
||||
assertEquals("/x is a directory not a file.", ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCreatingFileOverDirectory() throws Exception {
|
||||
Path dir = new Path("/x");
|
||||
assertTrue(fs.mkdirs(dir));
|
||||
try {
|
||||
fs.create(dir).close();
|
||||
assertTrue("Should've thrown", false);
|
||||
} catch (IOException ex) {
|
||||
assertEquals("Cannot create file /x; already exists as a directory.",
|
||||
ex.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetPermissionOnFile() throws Exception {
|
||||
Path newFile = new Path("testPermission");
|
||||
OutputStream output = fs.create(newFile);
|
||||
output.write(13);
|
||||
output.close();
|
||||
FsPermission newPermission = new FsPermission((short) 0700);
|
||||
fs.setPermission(newFile, newPermission);
|
||||
FileStatus newStatus = fs.getFileStatus(newFile);
|
||||
assertNotNull(newStatus);
|
||||
assertEquals(newPermission, newStatus.getPermission());
|
||||
assertEquals("supergroup", newStatus.getGroup());
|
||||
assertEquals(UserGroupInformation.getCurrentUser().getShortUserName(),
|
||||
newStatus.getOwner());
|
||||
assertEquals(1, newStatus.getLen());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetPermissionOnFolder() throws Exception {
|
||||
Path newFolder = new Path("testPermission");
|
||||
assertTrue(fs.mkdirs(newFolder));
|
||||
FsPermission newPermission = new FsPermission((short) 0600);
|
||||
fs.setPermission(newFolder, newPermission);
|
||||
FileStatus newStatus = fs.getFileStatus(newFolder);
|
||||
assertNotNull(newStatus);
|
||||
assertEquals(newPermission, newStatus.getPermission());
|
||||
assertTrue(newStatus.isDirectory());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetOwnerOnFile() throws Exception {
|
||||
Path newFile = new Path("testOwner");
|
||||
OutputStream output = fs.create(newFile);
|
||||
output.write(13);
|
||||
output.close();
|
||||
fs.setOwner(newFile, "newUser", null);
|
||||
FileStatus newStatus = fs.getFileStatus(newFile);
|
||||
assertNotNull(newStatus);
|
||||
assertEquals("newUser", newStatus.getOwner());
|
||||
assertEquals("supergroup", newStatus.getGroup());
|
||||
assertEquals(1, newStatus.getLen());
|
||||
fs.setOwner(newFile, null, "newGroup");
|
||||
newStatus = fs.getFileStatus(newFile);
|
||||
assertNotNull(newStatus);
|
||||
assertEquals("newUser", newStatus.getOwner());
|
||||
assertEquals("newGroup", newStatus.getGroup());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetOwnerOnFolder() throws Exception {
|
||||
Path newFolder = new Path("testOwner");
|
||||
assertTrue(fs.mkdirs(newFolder));
|
||||
fs.setOwner(newFolder, "newUser", null);
|
||||
FileStatus newStatus = fs.getFileStatus(newFolder);
|
||||
assertNotNull(newStatus);
|
||||
assertEquals("newUser", newStatus.getOwner());
|
||||
assertTrue(newStatus.isDirectory());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testModifiedTimeForFile() throws Exception {
|
||||
Path testFile = new Path("testFile");
|
||||
fs.create(testFile).close();
|
||||
testModifiedTime(testFile);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testModifiedTimeForFolder() throws Exception {
|
||||
Path testFolder = new Path("testFolder");
|
||||
assertTrue(fs.mkdirs(testFolder));
|
||||
testModifiedTime(testFolder);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFolderLastModifiedTime() throws Exception {
|
||||
Path parentFolder = new Path("testFolder");
|
||||
Path innerFile = new Path(parentFolder, "innerfile");
|
||||
assertTrue(fs.mkdirs(parentFolder));
|
||||
|
||||
// Create file
|
||||
long lastModifiedTime = fs.getFileStatus(parentFolder)
|
||||
.getModificationTime();
|
||||
// Wait at least the error margin
|
||||
Thread.sleep(modifiedTimeErrorMargin + 1);
|
||||
assertTrue(fs.createNewFile(innerFile));
|
||||
// The parent folder last modified time should have changed because we
|
||||
// create an inner file.
|
||||
assertFalse(testModifiedTime(parentFolder, lastModifiedTime));
|
||||
testModifiedTime(parentFolder);
|
||||
|
||||
// Rename file
|
||||
lastModifiedTime = fs.getFileStatus(parentFolder).getModificationTime();
|
||||
Path destFolder = new Path("testDestFolder");
|
||||
assertTrue(fs.mkdirs(destFolder));
|
||||
long destLastModifiedTime = fs.getFileStatus(destFolder)
|
||||
.getModificationTime();
|
||||
Thread.sleep(modifiedTimeErrorMargin + 1);
|
||||
Path destFile = new Path(destFolder, "innerfile");
|
||||
assertTrue(fs.rename(innerFile, destFile));
|
||||
// Both source and destination folder last modified time should have changed
|
||||
// because of renaming.
|
||||
assertFalse(testModifiedTime(parentFolder, lastModifiedTime));
|
||||
assertFalse(testModifiedTime(destFolder, destLastModifiedTime));
|
||||
testModifiedTime(parentFolder);
|
||||
testModifiedTime(destFolder);
|
||||
|
||||
// Delete file
|
||||
destLastModifiedTime = fs.getFileStatus(destFolder).getModificationTime();
|
||||
// Wait at least the error margin
|
||||
Thread.sleep(modifiedTimeErrorMargin + 1);
|
||||
fs.delete(destFile, false);
|
||||
// The parent folder last modified time should have changed because we
|
||||
// delete an inner file.
|
||||
assertFalse(testModifiedTime(destFolder, destLastModifiedTime));
|
||||
testModifiedTime(destFolder);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testListSlash() throws Exception {
|
||||
Path testFolder = new Path("/testFolder");
|
||||
Path testFile = new Path(testFolder, "testFile");
|
||||
assertTrue(fs.mkdirs(testFolder));
|
||||
assertTrue(fs.createNewFile(testFile));
|
||||
FileStatus status = fs.getFileStatus(new Path("/testFolder/."));
|
||||
assertNotNull(status);
|
||||
}
|
||||
|
||||
private boolean testModifiedTime(Path testPath, long time) throws Exception {
|
||||
FileStatus fileStatus = fs.getFileStatus(testPath);
|
||||
final long errorMargin = modifiedTimeErrorMargin;
|
||||
long lastModified = fileStatus.getModificationTime();
|
||||
return (lastModified > (time - errorMargin) && lastModified < (time + errorMargin));
|
||||
}
|
||||
|
||||
private void testModifiedTime(Path testPath) throws Exception {
|
||||
Calendar utc = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
|
||||
long currentUtcTime = utc.getTime().getTime();
|
||||
FileStatus fileStatus = fs.getFileStatus(testPath);
|
||||
assertTrue("Modification time "
|
||||
+ new Date(fileStatus.getModificationTime()) + " is not close to now: "
|
||||
+ utc.getTime(), testModifiedTime(testPath, currentUtcTime));
|
||||
}
|
||||
|
||||
private void createEmptyFile(Path testFile, FsPermission permission)
|
||||
throws IOException {
|
||||
FSDataOutputStream outputStream = fs.create(testFile, permission, true,
|
||||
4096, (short) 1, 1024, null);
|
||||
outputStream.close();
|
||||
}
|
||||
|
||||
private String readString(Path testFile) throws IOException {
|
||||
return readString(fs, testFile);
|
||||
}
|
||||
|
||||
private String readString(FileSystem fs, Path testFile) throws IOException {
|
||||
FSDataInputStream inputStream = fs.open(testFile);
|
||||
String ret = readString(inputStream);
|
||||
inputStream.close();
|
||||
return ret;
|
||||
}
|
||||
|
||||
private String readString(FSDataInputStream inputStream) throws IOException {
|
||||
BufferedReader reader = new BufferedReader(new InputStreamReader(
|
||||
inputStream));
|
||||
final int BUFFER_SIZE = 1024;
|
||||
char[] buffer = new char[BUFFER_SIZE];
|
||||
int count = reader.read(buffer, 0, BUFFER_SIZE);
|
||||
if (count >= BUFFER_SIZE) {
|
||||
throw new IOException("Exceeded buffer size");
|
||||
}
|
||||
inputStream.close();
|
||||
return new String(buffer, 0, count);
|
||||
}
|
||||
|
||||
private void writeString(Path path, String value) throws IOException {
|
||||
writeString(fs, path, value);
|
||||
}
|
||||
|
||||
private void writeString(FileSystem fs, Path path, String value)
|
||||
throws IOException {
|
||||
FSDataOutputStream outputStream = fs.create(path, true);
|
||||
writeString(outputStream, value);
|
||||
outputStream.close();
|
||||
}
|
||||
|
||||
private void writeString(FSDataOutputStream outputStream, String value)
|
||||
throws IOException {
|
||||
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
|
||||
outputStream));
|
||||
writer.write(value);
|
||||
writer.close();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,194 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.fail;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.DataInputStream;
|
||||
import java.io.DataOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.apache.hadoop.fs.permission.FsPermission;
|
||||
import org.apache.hadoop.fs.permission.PermissionStatus;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
public class TestAzureConcurrentOutOfBandIo {
|
||||
|
||||
// Class constants.
|
||||
static final int DOWNLOAD_BLOCK_SIZE = 8 * 1024 * 1024;
|
||||
static final int UPLOAD_BLOCK_SIZE = 4 * 1024 * 1024;
|
||||
static final int BLOB_SIZE = 32 * 1024 * 1024;
|
||||
|
||||
// Number of blocks to be written before flush.
|
||||
private static final int NUMBER_OF_BLOCKS = 2;
|
||||
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
// Overridden TestCase methods.
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createOutOfBandStore(
|
||||
UPLOAD_BLOCK_SIZE, DOWNLOAD_BLOCK_SIZE);
|
||||
assumeNotNull(testAccount);
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
}
|
||||
}
|
||||
|
||||
class DataBlockWriter implements Runnable {
|
||||
|
||||
Thread runner;
|
||||
AzureBlobStorageTestAccount writerStorageAccount;
|
||||
String key;
|
||||
boolean done = false;
|
||||
|
||||
/**
|
||||
* Constructor captures the test account.
|
||||
*
|
||||
* @param testAccount
|
||||
*/
|
||||
public DataBlockWriter(AzureBlobStorageTestAccount testAccount, String key) {
|
||||
writerStorageAccount = testAccount;
|
||||
this.key = key;
|
||||
}
|
||||
|
||||
/**
|
||||
* Start writing blocks to Azure storage.
|
||||
*/
|
||||
public void startWriting() {
|
||||
runner = new Thread(this); // Create the block writer thread.
|
||||
runner.start(); // Start the block writer thread.
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop writing blocks to Azure storage.
|
||||
*/
|
||||
public void stopWriting() {
|
||||
done = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Implementation of the runnable interface. The run method is a tight loop
|
||||
* which repeatedly updates the blob with a 4 MB block.
|
||||
*/
|
||||
public void run() {
|
||||
byte[] dataBlockWrite = new byte[UPLOAD_BLOCK_SIZE];
|
||||
|
||||
DataOutputStream outputStream = null;
|
||||
|
||||
try {
|
||||
for (int i = 0; !done; i++) {
|
||||
// Write two 4 MB blocks to the blob.
|
||||
//
|
||||
outputStream = writerStorageAccount.getStore().storefile(key,
|
||||
new PermissionStatus("", "", FsPermission.getDefault()));
|
||||
|
||||
Arrays.fill(dataBlockWrite, (byte) (i % 256));
|
||||
for (int j = 0; j < NUMBER_OF_BLOCKS; j++) {
|
||||
outputStream.write(dataBlockWrite);
|
||||
}
|
||||
|
||||
outputStream.flush();
|
||||
outputStream.close();
|
||||
}
|
||||
} catch (AzureException e) {
|
||||
System.out
|
||||
.println("DatablockWriter thread encountered a storage exception."
|
||||
+ e.getMessage());
|
||||
} catch (IOException e) {
|
||||
System.out
|
||||
.println("DatablockWriter thread encountered an I/O exception."
|
||||
+ e.getMessage());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testReadOOBWrites() throws Exception {
|
||||
|
||||
byte[] dataBlockWrite = new byte[UPLOAD_BLOCK_SIZE];
|
||||
byte[] dataBlockRead = new byte[UPLOAD_BLOCK_SIZE];
|
||||
|
||||
// Write to blob to make sure it exists.
|
||||
//
|
||||
// Write five 4 MB blocks to the blob. To ensure there is data in the blob
|
||||
// before reading. This eliminates the race between the reader and writer
|
||||
// threads.
|
||||
DataOutputStream outputStream = testAccount.getStore().storefile(
|
||||
"WASB_String.txt",
|
||||
new PermissionStatus("", "", FsPermission.getDefault()));
|
||||
Arrays.fill(dataBlockWrite, (byte) 255);
|
||||
for (int i = 0; i < NUMBER_OF_BLOCKS; i++) {
|
||||
outputStream.write(dataBlockWrite);
|
||||
}
|
||||
|
||||
outputStream.flush();
|
||||
outputStream.close();
|
||||
|
||||
// Start writing blocks to Azure store using the DataBlockWriter thread.
|
||||
DataBlockWriter writeBlockTask = new DataBlockWriter(testAccount,
|
||||
"WASB_String.txt");
|
||||
writeBlockTask.startWriting();
|
||||
int count = 0;
|
||||
DataInputStream inputStream = null;
|
||||
|
||||
for (int i = 0; i < 5; i++) {
|
||||
try {
|
||||
inputStream = testAccount.getStore().retrieve("WASB_String.txt", 0);
|
||||
count = 0;
|
||||
int c = 0;
|
||||
|
||||
while (c >= 0) {
|
||||
c = inputStream.read(dataBlockRead, 0, UPLOAD_BLOCK_SIZE);
|
||||
if (c < 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
// Counting the number of bytes.
|
||||
count += c;
|
||||
}
|
||||
} catch (IOException e) {
|
||||
System.out.println(e.getCause().toString());
|
||||
e.printStackTrace();
|
||||
fail();
|
||||
}
|
||||
|
||||
// Close the stream.
|
||||
if (null != inputStream) {
|
||||
inputStream.close();
|
||||
}
|
||||
}
|
||||
|
||||
// Stop writing blocks.
|
||||
writeBlockTask.stopWriting();
|
||||
|
||||
// Validate that a block was read.
|
||||
assertEquals(NUMBER_OF_BLOCKS * UPLOAD_BLOCK_SIZE, count);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,257 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.HttpURLConnection;
|
||||
import java.net.URI;
|
||||
import java.util.Arrays;
|
||||
import java.util.HashMap;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.FSDataInputStream;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.TestHookOperationContext;
|
||||
import org.apache.hadoop.fs.permission.FsPermission;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.SendingRequestEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageEvent;
|
||||
|
||||
public class TestAzureFileSystemErrorConditions {
|
||||
private static final int ALL_THREE_FILE_SIZE = 1024;
|
||||
|
||||
@Test
|
||||
public void testNoInitialize() throws Exception {
|
||||
AzureNativeFileSystemStore store = new AzureNativeFileSystemStore();
|
||||
boolean passed = false;
|
||||
try {
|
||||
store.retrieveMetadata("foo");
|
||||
passed = true;
|
||||
} catch (AssertionError e) {
|
||||
}
|
||||
assertFalse(
|
||||
"Doing an operation on the store should throw if not initalized.",
|
||||
passed);
|
||||
}
|
||||
|
||||
/**
|
||||
* Try accessing an unauthorized or non-existent (treated the same) container
|
||||
* from WASB.
|
||||
*/
|
||||
@Test
|
||||
public void testAccessUnauthorizedPublicContainer() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
AzureBlobStorageTestAccount.addWasbToConfiguration(conf);
|
||||
Path noAccessPath = new Path(
|
||||
"wasb://nonExistentContainer@hopefullyNonExistentAccount/someFile");
|
||||
NativeAzureFileSystem.suppressRetryPolicy();
|
||||
try {
|
||||
FileSystem.get(noAccessPath.toUri(), conf).open(noAccessPath);
|
||||
assertTrue("Should've thrown.", false);
|
||||
} catch (AzureException ex) {
|
||||
assertTrue("Unexpected message in exception " + ex, ex.getMessage()
|
||||
.contains(
|
||||
"Unable to access container nonExistentContainer in account"
|
||||
+ " hopefullyNonExistentAccount"));
|
||||
} finally {
|
||||
NativeAzureFileSystem.resumeRetryPolicy();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testAccessContainerWithWrongVersion() throws Exception {
|
||||
AzureNativeFileSystemStore store = new AzureNativeFileSystemStore();
|
||||
MockStorageInterface mockStorage = new MockStorageInterface();
|
||||
store.setAzureStorageInteractionLayer(mockStorage);
|
||||
FileSystem fs = new NativeAzureFileSystem(store);
|
||||
try {
|
||||
Configuration conf = new Configuration();
|
||||
AzureBlobStorageTestAccount.setMockAccountKey(conf);
|
||||
HashMap<String, String> metadata = new HashMap<String, String>();
|
||||
metadata.put(AzureNativeFileSystemStore.VERSION_METADATA_KEY,
|
||||
"2090-04-05"); // It's from the future!
|
||||
mockStorage.addPreExistingContainer(
|
||||
AzureBlobStorageTestAccount.getMockContainerUri(), metadata);
|
||||
|
||||
boolean passed = false;
|
||||
try {
|
||||
fs.initialize(new URI(AzureBlobStorageTestAccount.MOCK_WASB_URI), conf);
|
||||
fs.listStatus(new Path("/"));
|
||||
passed = true;
|
||||
} catch (AzureException ex) {
|
||||
assertTrue("Unexpected exception message: " + ex, ex.getMessage()
|
||||
.contains("unsupported version: 2090-04-05."));
|
||||
}
|
||||
assertFalse(
|
||||
"Should've thrown an exception because of the wrong version.", passed);
|
||||
} finally {
|
||||
fs.close();
|
||||
}
|
||||
}
|
||||
|
||||
private interface ConnectionRecognizer {
|
||||
boolean isTargetConnection(HttpURLConnection connection);
|
||||
}
|
||||
|
||||
private class TransientErrorInjector extends
|
||||
StorageEvent<SendingRequestEvent> {
|
||||
final ConnectionRecognizer connectionRecognizer;
|
||||
private boolean injectedErrorOnce = false;
|
||||
|
||||
public TransientErrorInjector(ConnectionRecognizer connectionRecognizer) {
|
||||
this.connectionRecognizer = connectionRecognizer;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void eventOccurred(SendingRequestEvent eventArg) {
|
||||
HttpURLConnection connection = (HttpURLConnection) eventArg
|
||||
.getConnectionObject();
|
||||
if (!connectionRecognizer.isTargetConnection(connection)) {
|
||||
return;
|
||||
}
|
||||
if (!injectedErrorOnce) {
|
||||
connection.setReadTimeout(1);
|
||||
connection.disconnect();
|
||||
injectedErrorOnce = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void injectTransientError(NativeAzureFileSystem fs,
|
||||
final ConnectionRecognizer connectionRecognizer) {
|
||||
fs.getStore().addTestHookToOperationContext(new TestHookOperationContext() {
|
||||
@Override
|
||||
public OperationContext modifyOperationContext(OperationContext original) {
|
||||
original.getSendingRequestEventHandler().addListener(
|
||||
new TransientErrorInjector(connectionRecognizer));
|
||||
return original;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTransientErrorOnDelete() throws Exception {
|
||||
// Need to do this test against a live storage account
|
||||
AzureBlobStorageTestAccount testAccount = AzureBlobStorageTestAccount
|
||||
.create();
|
||||
assumeNotNull(testAccount);
|
||||
try {
|
||||
NativeAzureFileSystem fs = testAccount.getFileSystem();
|
||||
injectTransientError(fs, new ConnectionRecognizer() {
|
||||
@Override
|
||||
public boolean isTargetConnection(HttpURLConnection connection) {
|
||||
return connection.getRequestMethod().equals("DELETE");
|
||||
}
|
||||
});
|
||||
Path testFile = new Path("/a/b");
|
||||
assertTrue(fs.createNewFile(testFile));
|
||||
assertTrue(fs.rename(testFile, new Path("/x")));
|
||||
} finally {
|
||||
testAccount.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
private void writeAllThreeFile(NativeAzureFileSystem fs, Path testFile)
|
||||
throws IOException {
|
||||
byte[] buffer = new byte[ALL_THREE_FILE_SIZE];
|
||||
Arrays.fill(buffer, (byte) 3);
|
||||
OutputStream stream = fs.create(testFile);
|
||||
stream.write(buffer);
|
||||
stream.close();
|
||||
}
|
||||
|
||||
private void readAllThreeFile(NativeAzureFileSystem fs, Path testFile)
|
||||
throws IOException {
|
||||
byte[] buffer = new byte[ALL_THREE_FILE_SIZE];
|
||||
InputStream inStream = fs.open(testFile);
|
||||
assertEquals(buffer.length, inStream.read(buffer, 0, buffer.length));
|
||||
inStream.close();
|
||||
for (int i = 0; i < buffer.length; i++) {
|
||||
assertEquals(3, buffer[i]);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTransientErrorOnCommitBlockList() throws Exception {
|
||||
// Need to do this test against a live storage account
|
||||
AzureBlobStorageTestAccount testAccount = AzureBlobStorageTestAccount
|
||||
.create();
|
||||
assumeNotNull(testAccount);
|
||||
try {
|
||||
NativeAzureFileSystem fs = testAccount.getFileSystem();
|
||||
injectTransientError(fs, new ConnectionRecognizer() {
|
||||
@Override
|
||||
public boolean isTargetConnection(HttpURLConnection connection) {
|
||||
return connection.getRequestMethod().equals("PUT")
|
||||
&& connection.getURL().getQuery().contains("blocklist");
|
||||
}
|
||||
});
|
||||
Path testFile = new Path("/a/b");
|
||||
writeAllThreeFile(fs, testFile);
|
||||
readAllThreeFile(fs, testFile);
|
||||
} finally {
|
||||
testAccount.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testTransientErrorOnRead() throws Exception {
|
||||
// Need to do this test against a live storage account
|
||||
AzureBlobStorageTestAccount testAccount = AzureBlobStorageTestAccount
|
||||
.create();
|
||||
assumeNotNull(testAccount);
|
||||
try {
|
||||
NativeAzureFileSystem fs = testAccount.getFileSystem();
|
||||
Path testFile = new Path("/a/b");
|
||||
writeAllThreeFile(fs, testFile);
|
||||
injectTransientError(fs, new ConnectionRecognizer() {
|
||||
@Override
|
||||
public boolean isTargetConnection(HttpURLConnection connection) {
|
||||
return connection.getRequestMethod().equals("GET");
|
||||
}
|
||||
});
|
||||
readAllThreeFile(fs, testFile);
|
||||
} finally {
|
||||
testAccount.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
// Tests an error during stream creation (in this case in the seek() implementation
|
||||
// to verify the close-stream-on-error logic.
|
||||
@Test (expected=AzureException.class)
|
||||
public void testErrorDuringRetrieve() throws Exception {
|
||||
NativeAzureFileSystem fs = AzureBlobStorageTestAccount.createMock().getFileSystem();
|
||||
Path testFile = new Path("/testErrorDuringRetrieve");
|
||||
writeAllThreeFile(fs, testFile);
|
||||
|
||||
FSDataInputStream stream = fs.open(testFile);
|
||||
stream.seek(Integer.MAX_VALUE);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,236 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.KEY_CHECK_BLOCK_MD5;
|
||||
import static org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.KEY_STORE_BLOB_MD5;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assert.fail;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.HttpURLConnection;
|
||||
import java.util.Arrays;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.TestHookOperationContext;
|
||||
import org.junit.After;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.microsoft.windowsazure.storage.Constants;
|
||||
import com.microsoft.windowsazure.storage.OperationContext;
|
||||
import com.microsoft.windowsazure.storage.ResponseReceivedEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageErrorCodeStrings;
|
||||
import com.microsoft.windowsazure.storage.StorageEvent;
|
||||
import com.microsoft.windowsazure.storage.StorageException;
|
||||
import com.microsoft.windowsazure.storage.blob.BlockEntry;
|
||||
import com.microsoft.windowsazure.storage.blob.BlockSearchMode;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
import com.microsoft.windowsazure.storage.core.Base64;
|
||||
|
||||
/**
|
||||
* Test that we do proper data integrity validation with MD5 checks as
|
||||
* configured.
|
||||
*/
|
||||
public class TestBlobDataValidation {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Test that by default we don't store the blob-level MD5.
|
||||
*/
|
||||
@Test
|
||||
public void testBlobMd5StoreOffByDefault() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create();
|
||||
testStoreBlobMd5(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Test that we get blob-level MD5 storage and validation if we specify that
|
||||
* in the configuration.
|
||||
*/
|
||||
@Test
|
||||
public void testStoreBlobMd5() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
conf.setBoolean(KEY_STORE_BLOB_MD5, true);
|
||||
testAccount = AzureBlobStorageTestAccount.create(conf);
|
||||
testStoreBlobMd5(true);
|
||||
}
|
||||
|
||||
private void testStoreBlobMd5(boolean expectMd5Stored) throws Exception {
|
||||
assumeNotNull(testAccount);
|
||||
// Write a test file.
|
||||
String testFileKey = "testFile";
|
||||
Path testFilePath = new Path("/" + testFileKey);
|
||||
OutputStream outStream = testAccount.getFileSystem().create(testFilePath);
|
||||
outStream.write(new byte[] { 5, 15 });
|
||||
outStream.close();
|
||||
|
||||
// Check that we stored/didn't store the MD5 field as configured.
|
||||
CloudBlockBlob blob = testAccount.getBlobReference(testFileKey);
|
||||
blob.downloadAttributes();
|
||||
String obtainedMd5 = blob.getProperties().getContentMD5();
|
||||
if (expectMd5Stored) {
|
||||
assertNotNull(obtainedMd5);
|
||||
} else {
|
||||
assertNull("Expected no MD5, found: " + obtainedMd5, obtainedMd5);
|
||||
}
|
||||
|
||||
// Mess with the content so it doesn't match the MD5.
|
||||
String newBlockId = Base64.encode(new byte[] { 55, 44, 33, 22 });
|
||||
blob.uploadBlock(newBlockId,
|
||||
new ByteArrayInputStream(new byte[] { 6, 45 }), 2);
|
||||
blob.commitBlockList(Arrays.asList(new BlockEntry[] { new BlockEntry(
|
||||
newBlockId, BlockSearchMode.UNCOMMITTED) }));
|
||||
|
||||
// Now read back the content. If we stored the MD5 for the blob content
|
||||
// we should get a data corruption error.
|
||||
InputStream inStream = testAccount.getFileSystem().open(testFilePath);
|
||||
try {
|
||||
byte[] inBuf = new byte[100];
|
||||
while (inStream.read(inBuf) > 0){
|
||||
//nothing;
|
||||
}
|
||||
inStream.close();
|
||||
if (expectMd5Stored) {
|
||||
fail("Should've thrown because of data corruption.");
|
||||
}
|
||||
} catch (IOException ex) {
|
||||
if (!expectMd5Stored) {
|
||||
throw ex;
|
||||
}
|
||||
StorageException cause = (StorageException) ex.getCause();
|
||||
assertNotNull(cause);
|
||||
assertTrue("Unexpected cause: " + cause,
|
||||
cause.getErrorCode().equals(StorageErrorCodeStrings.INVALID_MD5));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Test that by default we check block-level MD5.
|
||||
*/
|
||||
@Test
|
||||
public void testCheckBlockMd5() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create();
|
||||
testCheckBlockMd5(true);
|
||||
}
|
||||
|
||||
/**
|
||||
* Test that we don't check block-level MD5 if we specify that in the
|
||||
* configuration.
|
||||
*/
|
||||
@Test
|
||||
public void testDontCheckBlockMd5() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
conf.setBoolean(KEY_CHECK_BLOCK_MD5, false);
|
||||
testAccount = AzureBlobStorageTestAccount.create(conf);
|
||||
testCheckBlockMd5(false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Connection inspector to check that MD5 fields for content is set/not set as
|
||||
* expected.
|
||||
*/
|
||||
private static class ContentMD5Checker extends
|
||||
StorageEvent<ResponseReceivedEvent> {
|
||||
private final boolean expectMd5;
|
||||
|
||||
public ContentMD5Checker(boolean expectMd5) {
|
||||
this.expectMd5 = expectMd5;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void eventOccurred(ResponseReceivedEvent eventArg) {
|
||||
HttpURLConnection connection = (HttpURLConnection) eventArg
|
||||
.getConnectionObject();
|
||||
if (isGetRange(connection)) {
|
||||
checkObtainedMd5(connection
|
||||
.getHeaderField(Constants.HeaderConstants.CONTENT_MD5));
|
||||
} else if (isPutBlock(connection)) {
|
||||
checkObtainedMd5(connection
|
||||
.getRequestProperty(Constants.HeaderConstants.CONTENT_MD5));
|
||||
}
|
||||
}
|
||||
|
||||
private void checkObtainedMd5(String obtainedMd5) {
|
||||
if (expectMd5) {
|
||||
assertNotNull(obtainedMd5);
|
||||
} else {
|
||||
assertNull("Expected no MD5, found: " + obtainedMd5, obtainedMd5);
|
||||
}
|
||||
}
|
||||
|
||||
private static boolean isPutBlock(HttpURLConnection connection) {
|
||||
return connection.getRequestMethod().equals("PUT")
|
||||
&& connection.getURL().getQuery().contains("blockid");
|
||||
}
|
||||
|
||||
private static boolean isGetRange(HttpURLConnection connection) {
|
||||
return connection.getRequestMethod().equals("GET")
|
||||
&& connection
|
||||
.getHeaderField(Constants.HeaderConstants.STORAGE_RANGE_HEADER) != null;
|
||||
}
|
||||
}
|
||||
|
||||
private void testCheckBlockMd5(final boolean expectMd5Checked)
|
||||
throws Exception {
|
||||
assumeNotNull(testAccount);
|
||||
Path testFilePath = new Path("/testFile");
|
||||
|
||||
// Add a hook to check that for GET/PUT requests we set/don't set
|
||||
// the block-level MD5 field as configured. I tried to do clever
|
||||
// testing by also messing with the raw data to see if we actually
|
||||
// validate the data as expected, but the HttpURLConnection wasn't
|
||||
// pluggable enough for me to do that.
|
||||
testAccount.getFileSystem().getStore()
|
||||
.addTestHookToOperationContext(new TestHookOperationContext() {
|
||||
@Override
|
||||
public OperationContext modifyOperationContext(
|
||||
OperationContext original) {
|
||||
original.getResponseReceivedEventHandler().addListener(
|
||||
new ContentMD5Checker(expectMd5Checked));
|
||||
return original;
|
||||
}
|
||||
});
|
||||
|
||||
OutputStream outStream = testAccount.getFileSystem().create(testFilePath);
|
||||
outStream.write(new byte[] { 5, 15 });
|
||||
outStream.close();
|
||||
|
||||
InputStream inStream = testAccount.getFileSystem().open(testFilePath);
|
||||
byte[] inBuf = new byte[100];
|
||||
while (inStream.read(inBuf) > 0){
|
||||
//nothing;
|
||||
}
|
||||
inStream.close();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,265 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertNull;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.util.HashMap;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.permission.FsAction;
|
||||
import org.apache.hadoop.fs.permission.FsPermission;
|
||||
import org.apache.hadoop.security.UserGroupInformation;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
/**
|
||||
* Tests that we put the correct metadata on blobs created through WASB.
|
||||
*/
|
||||
public class TestBlobMetadata {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
private FileSystem fs;
|
||||
private InMemoryBlockBlobStore backingStore;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
fs = testAccount.getFileSystem();
|
||||
backingStore = testAccount.getMockStorage().getBackingStore();
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
testAccount.cleanup();
|
||||
fs = null;
|
||||
backingStore = null;
|
||||
}
|
||||
|
||||
private static String getExpectedOwner() throws Exception {
|
||||
return UserGroupInformation.getCurrentUser().getShortUserName();
|
||||
}
|
||||
|
||||
private static String getExpectedPermissionString(String permissionString)
|
||||
throws Exception {
|
||||
return String.format(
|
||||
"{\"owner\":\"%s\",\"group\":\"%s\",\"permissions\":\"%s\"}",
|
||||
getExpectedOwner(), NativeAzureFileSystem.AZURE_DEFAULT_GROUP_DEFAULT,
|
||||
permissionString);
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that WASB stamped the version in the container metadata.
|
||||
*/
|
||||
@Test
|
||||
public void testContainerVersionMetadata() throws Exception {
|
||||
// Do a write operation to trigger version stamp
|
||||
fs.createNewFile(new Path("/foo"));
|
||||
HashMap<String, String> containerMetadata = backingStore
|
||||
.getContainerMetadata();
|
||||
assertNotNull(containerMetadata);
|
||||
assertEquals(AzureNativeFileSystemStore.CURRENT_WASB_VERSION,
|
||||
containerMetadata.get(AzureNativeFileSystemStore.VERSION_METADATA_KEY));
|
||||
}
|
||||
|
||||
private static final class FsWithPreExistingContainer implements Closeable {
|
||||
private final MockStorageInterface mockStorage;
|
||||
private final NativeAzureFileSystem fs;
|
||||
|
||||
private FsWithPreExistingContainer(MockStorageInterface mockStorage,
|
||||
NativeAzureFileSystem fs) {
|
||||
this.mockStorage = mockStorage;
|
||||
this.fs = fs;
|
||||
}
|
||||
|
||||
public NativeAzureFileSystem getFs() {
|
||||
return fs;
|
||||
}
|
||||
|
||||
public HashMap<String, String> getContainerMetadata() {
|
||||
return mockStorage.getBackingStore().getContainerMetadata();
|
||||
}
|
||||
|
||||
public static FsWithPreExistingContainer create() throws Exception {
|
||||
return create(null);
|
||||
}
|
||||
|
||||
public static FsWithPreExistingContainer create(
|
||||
HashMap<String, String> containerMetadata) throws Exception {
|
||||
AzureNativeFileSystemStore store = new AzureNativeFileSystemStore();
|
||||
MockStorageInterface mockStorage = new MockStorageInterface();
|
||||
store.setAzureStorageInteractionLayer(mockStorage);
|
||||
NativeAzureFileSystem fs = new NativeAzureFileSystem(store);
|
||||
Configuration conf = new Configuration();
|
||||
AzureBlobStorageTestAccount.setMockAccountKey(conf);
|
||||
mockStorage.addPreExistingContainer(
|
||||
AzureBlobStorageTestAccount.getMockContainerUri(), containerMetadata);
|
||||
fs.initialize(new URI(AzureBlobStorageTestAccount.MOCK_WASB_URI), conf);
|
||||
return new FsWithPreExistingContainer(mockStorage, fs);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() throws IOException {
|
||||
fs.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that WASB stamped the version in the container metadata if it does a
|
||||
* write operation to a pre-existing container.
|
||||
*/
|
||||
@Test
|
||||
public void testPreExistingContainerVersionMetadata() throws Exception {
|
||||
// Create a mock storage with a pre-existing container that has no
|
||||
// WASB version metadata on it.
|
||||
FsWithPreExistingContainer fsWithContainer = FsWithPreExistingContainer
|
||||
.create();
|
||||
|
||||
// Now, do some read operations (should touch the metadata)
|
||||
assertFalse(fsWithContainer.getFs().exists(new Path("/IDontExist")));
|
||||
assertEquals(0, fsWithContainer.getFs().listStatus(new Path("/")).length);
|
||||
|
||||
// Check that no container metadata exists yet
|
||||
assertNull(fsWithContainer.getContainerMetadata());
|
||||
|
||||
// Now do a write operation - should stamp the version
|
||||
fsWithContainer.getFs().mkdirs(new Path("/dir"));
|
||||
|
||||
// Check that now we have the version stamp
|
||||
assertNotNull(fsWithContainer.getContainerMetadata());
|
||||
assertEquals(
|
||||
AzureNativeFileSystemStore.CURRENT_WASB_VERSION,
|
||||
fsWithContainer.getContainerMetadata().get(
|
||||
AzureNativeFileSystemStore.VERSION_METADATA_KEY));
|
||||
fsWithContainer.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that WASB works well with an older version container with ASV-era
|
||||
* version and metadata.
|
||||
*/
|
||||
@Test
|
||||
public void testFirstContainerVersionMetadata() throws Exception {
|
||||
// Create a mock storage with a pre-existing container that has
|
||||
// ASV version metadata on it.
|
||||
HashMap<String, String> containerMetadata = new HashMap<String, String>();
|
||||
containerMetadata.put(AzureNativeFileSystemStore.OLD_VERSION_METADATA_KEY,
|
||||
AzureNativeFileSystemStore.FIRST_WASB_VERSION);
|
||||
FsWithPreExistingContainer fsWithContainer = FsWithPreExistingContainer
|
||||
.create(containerMetadata);
|
||||
|
||||
// Now, do some read operations (should touch the metadata)
|
||||
assertFalse(fsWithContainer.getFs().exists(new Path("/IDontExist")));
|
||||
assertEquals(0, fsWithContainer.getFs().listStatus(new Path("/")).length);
|
||||
|
||||
// Check that no container metadata exists yet
|
||||
assertEquals(
|
||||
AzureNativeFileSystemStore.FIRST_WASB_VERSION,
|
||||
fsWithContainer.getContainerMetadata().get(
|
||||
AzureNativeFileSystemStore.OLD_VERSION_METADATA_KEY));
|
||||
assertNull(fsWithContainer.getContainerMetadata().get(
|
||||
AzureNativeFileSystemStore.VERSION_METADATA_KEY));
|
||||
|
||||
// Now do a write operation - should stamp the version
|
||||
fsWithContainer.getFs().mkdirs(new Path("/dir"));
|
||||
|
||||
// Check that now we have the version stamp
|
||||
assertEquals(
|
||||
AzureNativeFileSystemStore.CURRENT_WASB_VERSION,
|
||||
fsWithContainer.getContainerMetadata().get(
|
||||
AzureNativeFileSystemStore.VERSION_METADATA_KEY));
|
||||
assertNull(fsWithContainer.getContainerMetadata().get(
|
||||
AzureNativeFileSystemStore.OLD_VERSION_METADATA_KEY));
|
||||
fsWithContainer.close();
|
||||
}
|
||||
|
||||
@SuppressWarnings("deprecation")
|
||||
@Test
|
||||
public void testPermissionMetadata() throws Exception {
|
||||
FsPermission justMe = new FsPermission(FsAction.READ_WRITE, FsAction.NONE,
|
||||
FsAction.NONE);
|
||||
Path selfishFile = new Path("/noOneElse");
|
||||
fs.create(selfishFile, justMe, true, 4096, fs.getDefaultReplication(),
|
||||
fs.getDefaultBlockSize(), null).close();
|
||||
HashMap<String, String> metadata = backingStore
|
||||
.getMetadata(AzureBlobStorageTestAccount.toMockUri(selfishFile));
|
||||
assertNotNull(metadata);
|
||||
String storedPermission = metadata.get("hdi_permission");
|
||||
assertEquals(getExpectedPermissionString("rw-------"), storedPermission);
|
||||
FileStatus retrievedStatus = fs.getFileStatus(selfishFile);
|
||||
assertNotNull(retrievedStatus);
|
||||
assertEquals(justMe, retrievedStatus.getPermission());
|
||||
assertEquals(getExpectedOwner(), retrievedStatus.getOwner());
|
||||
assertEquals(NativeAzureFileSystem.AZURE_DEFAULT_GROUP_DEFAULT,
|
||||
retrievedStatus.getGroup());
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that WASB understands the old-style ASV metadata and changes it when
|
||||
* it gets the chance.
|
||||
*/
|
||||
@Test
|
||||
public void testOldPermissionMetadata() throws Exception {
|
||||
Path selfishFile = new Path("/noOneElse");
|
||||
HashMap<String, String> metadata = new HashMap<String, String>();
|
||||
metadata.put("asv_permission", getExpectedPermissionString("rw-------"));
|
||||
backingStore.setContent(AzureBlobStorageTestAccount.toMockUri(selfishFile),
|
||||
new byte[] {}, metadata);
|
||||
FsPermission justMe = new FsPermission(FsAction.READ_WRITE, FsAction.NONE,
|
||||
FsAction.NONE);
|
||||
FileStatus retrievedStatus = fs.getFileStatus(selfishFile);
|
||||
assertNotNull(retrievedStatus);
|
||||
assertEquals(justMe, retrievedStatus.getPermission());
|
||||
assertEquals(getExpectedOwner(), retrievedStatus.getOwner());
|
||||
assertEquals(NativeAzureFileSystem.AZURE_DEFAULT_GROUP_DEFAULT,
|
||||
retrievedStatus.getGroup());
|
||||
FsPermission meAndYou = new FsPermission(FsAction.READ_WRITE,
|
||||
FsAction.READ_WRITE, FsAction.NONE);
|
||||
fs.setPermission(selfishFile, meAndYou);
|
||||
metadata = backingStore.getMetadata(AzureBlobStorageTestAccount
|
||||
.toMockUri(selfishFile));
|
||||
assertNotNull(metadata);
|
||||
String storedPermission = metadata.get("hdi_permission");
|
||||
assertEquals(getExpectedPermissionString("rw-rw----"), storedPermission);
|
||||
assertNull(metadata.get("asv_permission"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFolderMetadata() throws Exception {
|
||||
Path folder = new Path("/folder");
|
||||
FsPermission justRead = new FsPermission(FsAction.READ, FsAction.READ,
|
||||
FsAction.READ);
|
||||
fs.mkdirs(folder, justRead);
|
||||
HashMap<String, String> metadata = backingStore
|
||||
.getMetadata(AzureBlobStorageTestAccount.toMockUri(folder));
|
||||
assertNotNull(metadata);
|
||||
assertEquals("true", metadata.get("hdi_isfolder"));
|
||||
assertEquals(getExpectedPermissionString("r--r--r--"),
|
||||
metadata.get("hdi_permission"));
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,175 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.FileNotFoundException;
|
||||
import java.util.EnumSet;
|
||||
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.CreateOptions;
|
||||
import org.junit.After;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.microsoft.windowsazure.storage.blob.BlobOutputStream;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobContainer;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
|
||||
/**
|
||||
* Tests that WASB creates containers only if needed.
|
||||
*/
|
||||
public class TestContainerChecks {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testContainerExistAfterDoesNotExist() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create("",
|
||||
EnumSet.noneOf(CreateOptions.class));
|
||||
assumeNotNull(testAccount);
|
||||
CloudBlobContainer container = testAccount.getRealContainer();
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
|
||||
// Starting off with the container not there
|
||||
assertFalse(container.exists());
|
||||
|
||||
// A list shouldn't create the container and will set file system store
|
||||
// state to DoesNotExist
|
||||
try {
|
||||
fs.listStatus(new Path("/"));
|
||||
assertTrue("Should've thrown.", false);
|
||||
} catch (FileNotFoundException ex) {
|
||||
assertTrue("Unexpected exception: " + ex,
|
||||
ex.getMessage().contains("does not exist."));
|
||||
}
|
||||
assertFalse(container.exists());
|
||||
|
||||
// Create a container outside of the WASB FileSystem
|
||||
container.create();
|
||||
// Add a file to the container outside of the WASB FileSystem
|
||||
CloudBlockBlob blob = testAccount.getBlobReference("foo");
|
||||
BlobOutputStream outputStream = blob.openOutputStream();
|
||||
outputStream.write(new byte[10]);
|
||||
outputStream.close();
|
||||
|
||||
// Make sure the file is visible
|
||||
assertTrue(fs.exists(new Path("/foo")));
|
||||
assertTrue(container.exists());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testContainerCreateAfterDoesNotExist() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create("",
|
||||
EnumSet.noneOf(CreateOptions.class));
|
||||
assumeNotNull(testAccount);
|
||||
CloudBlobContainer container = testAccount.getRealContainer();
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
|
||||
// Starting off with the container not there
|
||||
assertFalse(container.exists());
|
||||
|
||||
// A list shouldn't create the container and will set file system store
|
||||
// state to DoesNotExist
|
||||
try {
|
||||
assertNull(fs.listStatus(new Path("/")));
|
||||
assertTrue("Should've thrown.", false);
|
||||
} catch (FileNotFoundException ex) {
|
||||
assertTrue("Unexpected exception: " + ex,
|
||||
ex.getMessage().contains("does not exist."));
|
||||
}
|
||||
assertFalse(container.exists());
|
||||
|
||||
// Create a container outside of the WASB FileSystem
|
||||
container.create();
|
||||
|
||||
// Write should succeed
|
||||
assertTrue(fs.createNewFile(new Path("/foo")));
|
||||
assertTrue(container.exists());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testContainerCreateOnWrite() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create("",
|
||||
EnumSet.noneOf(CreateOptions.class));
|
||||
assumeNotNull(testAccount);
|
||||
CloudBlobContainer container = testAccount.getRealContainer();
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
|
||||
// Starting off with the container not there
|
||||
assertFalse(container.exists());
|
||||
|
||||
// A list shouldn't create the container.
|
||||
try {
|
||||
fs.listStatus(new Path("/"));
|
||||
assertTrue("Should've thrown.", false);
|
||||
} catch (FileNotFoundException ex) {
|
||||
assertTrue("Unexpected exception: " + ex,
|
||||
ex.getMessage().contains("does not exist."));
|
||||
}
|
||||
assertFalse(container.exists());
|
||||
|
||||
// Neither should a read.
|
||||
try {
|
||||
fs.open(new Path("/foo"));
|
||||
assertFalse("Should've thrown.", true);
|
||||
} catch (FileNotFoundException ex) {
|
||||
}
|
||||
assertFalse(container.exists());
|
||||
|
||||
// Neither should a rename
|
||||
assertFalse(fs.rename(new Path("/foo"), new Path("/bar")));
|
||||
assertFalse(container.exists());
|
||||
|
||||
// But a write should.
|
||||
assertTrue(fs.createNewFile(new Path("/foo")));
|
||||
assertTrue(container.exists());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testContainerChecksWithSas() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create("",
|
||||
EnumSet.of(CreateOptions.UseSas));
|
||||
assumeNotNull(testAccount);
|
||||
CloudBlobContainer container = testAccount.getRealContainer();
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
|
||||
// The container shouldn't be there
|
||||
assertFalse(container.exists());
|
||||
|
||||
// A write should just fail
|
||||
try {
|
||||
fs.createNewFile(new Path("/foo"));
|
||||
assertFalse("Should've thrown.", true);
|
||||
} catch (AzureException ex) {
|
||||
}
|
||||
assertFalse(container.exists());
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,139 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
|
||||
import java.io.OutputStream;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.BlockLocation;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.junit.Test;
|
||||
|
||||
public class TestNativeAzureFileSystemBlockLocations {
|
||||
@Test
|
||||
public void testNumberOfBlocks() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
conf.set(NativeAzureFileSystem.AZURE_BLOCK_SIZE_PROPERTY_NAME, "500");
|
||||
AzureBlobStorageTestAccount testAccount = AzureBlobStorageTestAccount
|
||||
.createMock(conf);
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
Path testFile = createTestFile(fs, 1200);
|
||||
FileStatus stat = fs.getFileStatus(testFile);
|
||||
assertEquals(500, stat.getBlockSize());
|
||||
testAccount.cleanup();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsTypical() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(210, 50, 0, 210);
|
||||
assertEquals(5, locations.length);
|
||||
assertEquals("localhost", locations[0].getHosts()[0]);
|
||||
assertEquals(50, locations[0].getLength());
|
||||
assertEquals(10, locations[4].getLength());
|
||||
assertEquals(100, locations[2].getOffset());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsEmptyFile() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(0, 50, 0, 0);
|
||||
assertEquals(0, locations.length);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsSmallFile() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(1, 50, 0, 1);
|
||||
assertEquals(1, locations.length);
|
||||
assertEquals(1, locations[0].getLength());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsExactBlockSizeMultiple() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(200, 50, 0, 200);
|
||||
assertEquals(4, locations.length);
|
||||
assertEquals(150, locations[3].getOffset());
|
||||
assertEquals(50, locations[3].getLength());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsSubsetOfFile() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(205, 10, 15, 35);
|
||||
assertEquals(4, locations.length);
|
||||
assertEquals(10, locations[0].getLength());
|
||||
assertEquals(15, locations[0].getOffset());
|
||||
assertEquals(5, locations[3].getLength());
|
||||
assertEquals(45, locations[3].getOffset());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsOutOfRangeSubsetOfFile() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(205, 10, 300, 10);
|
||||
assertEquals(0, locations.length);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsEmptySubsetOfFile() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(205, 10, 0, 0);
|
||||
assertEquals(0, locations.length);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBlockLocationsDifferentLocationHost() throws Exception {
|
||||
BlockLocation[] locations = getBlockLocationsOutput(100, 10, 0, 100,
|
||||
"myblobhost");
|
||||
assertEquals(10, locations.length);
|
||||
assertEquals("myblobhost", locations[0].getHosts()[0]);
|
||||
}
|
||||
|
||||
private static BlockLocation[] getBlockLocationsOutput(int fileSize,
|
||||
int blockSize, long start, long len) throws Exception {
|
||||
return getBlockLocationsOutput(fileSize, blockSize, start, len, null);
|
||||
}
|
||||
|
||||
private static BlockLocation[] getBlockLocationsOutput(int fileSize,
|
||||
int blockSize, long start, long len, String blockLocationHost)
|
||||
throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
conf.set(NativeAzureFileSystem.AZURE_BLOCK_SIZE_PROPERTY_NAME, ""
|
||||
+ blockSize);
|
||||
if (blockLocationHost != null) {
|
||||
conf.set(NativeAzureFileSystem.AZURE_BLOCK_LOCATION_HOST_PROPERTY_NAME,
|
||||
blockLocationHost);
|
||||
}
|
||||
AzureBlobStorageTestAccount testAccount = AzureBlobStorageTestAccount
|
||||
.createMock(conf);
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
Path testFile = createTestFile(fs, fileSize);
|
||||
FileStatus stat = fs.getFileStatus(testFile);
|
||||
BlockLocation[] locations = fs.getFileBlockLocations(stat, start, len);
|
||||
testAccount.cleanup();
|
||||
return locations;
|
||||
}
|
||||
|
||||
private static Path createTestFile(FileSystem fs, int size) throws Exception {
|
||||
Path testFile = new Path("/testFile");
|
||||
OutputStream outputStream = fs.create(testFile);
|
||||
outputStream.write(new byte[size]);
|
||||
outputStream.close();
|
||||
return testFile;
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,186 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
|
||||
import java.io.OutputStream;
|
||||
import java.io.PrintWriter;
|
||||
import java.io.StringWriter;
|
||||
import java.util.HashMap;
|
||||
import java.util.Iterator;
|
||||
import java.util.concurrent.ConcurrentLinkedQueue;
|
||||
|
||||
import org.apache.hadoop.fs.FSDataOutputStream;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.util.StringUtils;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
public class TestNativeAzureFileSystemConcurrency {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
private FileSystem fs;
|
||||
private InMemoryBlockBlobStore backingStore;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
fs = testAccount.getFileSystem();
|
||||
backingStore = testAccount.getMockStorage().getBackingStore();
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
testAccount.cleanup();
|
||||
fs = null;
|
||||
backingStore = null;
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testLinkBlobs() throws Exception {
|
||||
Path filePath = new Path("/inProgress");
|
||||
FSDataOutputStream outputStream = fs.create(filePath);
|
||||
// Since the stream is still open, we should see an empty link
|
||||
// blob in the backing store linking to the temporary file.
|
||||
HashMap<String, String> metadata = backingStore
|
||||
.getMetadata(AzureBlobStorageTestAccount.toMockUri(filePath));
|
||||
assertNotNull(metadata);
|
||||
String linkValue = metadata
|
||||
.get(AzureNativeFileSystemStore.LINK_BACK_TO_UPLOAD_IN_PROGRESS_METADATA_KEY);
|
||||
assertNotNull(linkValue);
|
||||
assertTrue(backingStore.exists(AzureBlobStorageTestAccount
|
||||
.toMockUri(linkValue)));
|
||||
// Also, WASB should say the file exists now even before we close the
|
||||
// stream.
|
||||
assertTrue(fs.exists(filePath));
|
||||
outputStream.close();
|
||||
// Now there should be no link metadata on the final file.
|
||||
metadata = backingStore.getMetadata(AzureBlobStorageTestAccount
|
||||
.toMockUri(filePath));
|
||||
assertNull(metadata
|
||||
.get(AzureNativeFileSystemStore.LINK_BACK_TO_UPLOAD_IN_PROGRESS_METADATA_KEY));
|
||||
}
|
||||
|
||||
private static String toString(FileStatus[] list) {
|
||||
String[] asStrings = new String[list.length];
|
||||
for (int i = 0; i < list.length; i++) {
|
||||
asStrings[i] = list[i].getPath().toString();
|
||||
}
|
||||
return StringUtils.join(",", asStrings);
|
||||
}
|
||||
|
||||
/**
|
||||
* Test to make sure that we don't expose the temporary upload folder when
|
||||
* listing at the root.
|
||||
*/
|
||||
@Test
|
||||
public void testNoTempBlobsVisible() throws Exception {
|
||||
Path filePath = new Path("/inProgress");
|
||||
FSDataOutputStream outputStream = fs.create(filePath);
|
||||
// Make sure I can't see the temporary blob if I ask for a listing
|
||||
FileStatus[] listOfRoot = fs.listStatus(new Path("/"));
|
||||
assertEquals("Expected one file listed, instead got: "
|
||||
+ toString(listOfRoot), 1, listOfRoot.length);
|
||||
assertEquals(fs.makeQualified(filePath), listOfRoot[0].getPath());
|
||||
outputStream.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a collection of exceptions to a collection of strings by getting
|
||||
* the stack trace on every exception.
|
||||
*/
|
||||
private static Iterable<String> selectToString(
|
||||
final Iterable<Throwable> collection) {
|
||||
return new Iterable<String>() {
|
||||
@Override
|
||||
public Iterator<String> iterator() {
|
||||
final Iterator<Throwable> exceptionIterator = collection.iterator();
|
||||
return new Iterator<String>() {
|
||||
@Override
|
||||
public boolean hasNext() {
|
||||
return exceptionIterator.hasNext();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String next() {
|
||||
StringWriter stringWriter = new StringWriter();
|
||||
PrintWriter printWriter = new PrintWriter(stringWriter);
|
||||
exceptionIterator.next().printStackTrace(printWriter);
|
||||
printWriter.close();
|
||||
return stringWriter.toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void remove() {
|
||||
exceptionIterator.remove();
|
||||
}
|
||||
};
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests running starting multiple threads all doing various File system
|
||||
* operations against the same FS.
|
||||
*/
|
||||
@Test
|
||||
public void testMultiThreadedOperation() throws Exception {
|
||||
for (int iter = 0; iter < 10; iter++) {
|
||||
final int numThreads = 20;
|
||||
Thread[] threads = new Thread[numThreads];
|
||||
final ConcurrentLinkedQueue<Throwable> exceptionsEncountered = new ConcurrentLinkedQueue<Throwable>();
|
||||
for (int i = 0; i < numThreads; i++) {
|
||||
final Path threadLocalFile = new Path("/myFile" + i);
|
||||
threads[i] = new Thread(new Runnable() {
|
||||
@Override
|
||||
public void run() {
|
||||
try {
|
||||
assertTrue(!fs.exists(threadLocalFile));
|
||||
OutputStream output = fs.create(threadLocalFile);
|
||||
output.write(5);
|
||||
output.close();
|
||||
assertTrue(fs.exists(threadLocalFile));
|
||||
assertTrue(fs.listStatus(new Path("/")).length > 0);
|
||||
} catch (Throwable ex) {
|
||||
exceptionsEncountered.add(ex);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
for (Thread t : threads) {
|
||||
t.start();
|
||||
}
|
||||
for (Thread t : threads) {
|
||||
t.join();
|
||||
}
|
||||
assertTrue(
|
||||
"Encountered exceptions: "
|
||||
+ StringUtils.join("\r\n", selectToString(exceptionsEncountered)),
|
||||
exceptionsEncountered.isEmpty());
|
||||
tearDown();
|
||||
setUp();
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,50 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.fs.FileSystemContractBaseTest;
|
||||
|
||||
public class TestNativeAzureFileSystemContractEmulator extends
|
||||
FileSystemContractBaseTest {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@Override
|
||||
protected void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createForEmulator();
|
||||
if (testAccount != null) {
|
||||
fs = testAccount.getFileSystem();
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
fs = null;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void runTest() throws Throwable {
|
||||
if (testAccount != null) {
|
||||
super.runTest();
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,50 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.fs.FileSystemContractBaseTest;
|
||||
|
||||
public class TestNativeAzureFileSystemContractLive extends
|
||||
FileSystemContractBaseTest {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@Override
|
||||
protected void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create();
|
||||
if (testAccount != null) {
|
||||
fs = testAccount.getFileSystem();
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
fs = null;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void runTest() throws Throwable {
|
||||
if (testAccount != null) {
|
||||
super.runTest();
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import org.apache.hadoop.fs.FileSystemContractBaseTest;
|
||||
|
||||
public class TestNativeAzureFileSystemContractMocked extends
|
||||
FileSystemContractBaseTest {
|
||||
|
||||
@Override
|
||||
protected void setUp() throws Exception {
|
||||
fs = AzureBlobStorageTestAccount.createMock().getFileSystem();
|
||||
}
|
||||
|
||||
}
|
||||
@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assert.fail;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.HashMap;
|
||||
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
/**
|
||||
* Tests the scenario where a colon is included in the file/directory name.
|
||||
*
|
||||
* NativeAzureFileSystem#create(), #mkdir(), and #rename() disallow the
|
||||
* creation/rename of files/directories through WASB that have colons in the
|
||||
* names.
|
||||
*/
|
||||
public class TestNativeAzureFileSystemFileNameCheck {
|
||||
private FileSystem fs = null;
|
||||
private AzureBlobStorageTestAccount testAccount = null;
|
||||
private String root = null;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
fs = testAccount.getFileSystem();
|
||||
root = fs.getUri().toString();
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
testAccount.cleanup();
|
||||
root = null;
|
||||
fs = null;
|
||||
testAccount = null;
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCreate() throws Exception {
|
||||
// positive test
|
||||
Path testFile1 = new Path(root + "/testFile1");
|
||||
assertTrue(fs.createNewFile(testFile1));
|
||||
|
||||
// negative test
|
||||
Path testFile2 = new Path(root + "/testFile2:2");
|
||||
try {
|
||||
fs.createNewFile(testFile2);
|
||||
fail("Should've thrown.");
|
||||
} catch (IOException e) { // ignore
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testRename() throws Exception {
|
||||
// positive test
|
||||
Path testFile1 = new Path(root + "/testFile1");
|
||||
assertTrue(fs.createNewFile(testFile1));
|
||||
Path testFile2 = new Path(root + "/testFile2");
|
||||
fs.rename(testFile1, testFile2);
|
||||
assertTrue(!fs.exists(testFile1) && fs.exists(testFile2));
|
||||
|
||||
// negative test
|
||||
Path testFile3 = new Path(root + "/testFile3:3");
|
||||
try {
|
||||
fs.rename(testFile2, testFile3);
|
||||
fail("Should've thrown.");
|
||||
} catch (IOException e) { // ignore
|
||||
}
|
||||
assertTrue(fs.exists(testFile2));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMkdirs() throws Exception {
|
||||
// positive test
|
||||
Path testFolder1 = new Path(root + "/testFolder1");
|
||||
assertTrue(fs.mkdirs(testFolder1));
|
||||
|
||||
// negative test
|
||||
Path testFolder2 = new Path(root + "/testFolder2:2");
|
||||
try {
|
||||
assertTrue(fs.mkdirs(testFolder2));
|
||||
fail("Should've thrown.");
|
||||
} catch (IOException e) { // ignore
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testWasbFsck() throws Exception {
|
||||
// positive test
|
||||
Path testFolder1 = new Path(root + "/testFolder1");
|
||||
assertTrue(fs.mkdirs(testFolder1));
|
||||
Path testFolder2 = new Path(testFolder1, "testFolder2");
|
||||
assertTrue(fs.mkdirs(testFolder2));
|
||||
Path testFolder3 = new Path(testFolder1, "testFolder3");
|
||||
assertTrue(fs.mkdirs(testFolder3));
|
||||
Path testFile1 = new Path(testFolder2, "testFile1");
|
||||
assertTrue(fs.createNewFile(testFile1));
|
||||
Path testFile2 = new Path(testFolder1, "testFile2");
|
||||
assertTrue(fs.createNewFile(testFile2));
|
||||
assertFalse(runWasbFsck(testFolder1));
|
||||
|
||||
// negative test
|
||||
InMemoryBlockBlobStore backingStore = testAccount.getMockStorage()
|
||||
.getBackingStore();
|
||||
backingStore.setContent(AzureBlobStorageTestAccount
|
||||
.toMockUri("testFolder1/testFolder2/test2:2"), new byte[] { 1, 2 },
|
||||
new HashMap<String, String>());
|
||||
assertTrue(runWasbFsck(testFolder1));
|
||||
}
|
||||
|
||||
private boolean runWasbFsck(Path p) throws Exception {
|
||||
WasbFsck fsck = new WasbFsck(fs.getConf());
|
||||
fsck.setMockFileSystemForTesting(fs);
|
||||
fsck.run(new String[] { p.toString() });
|
||||
return fsck.getPathNameWarning();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,32 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
/*
|
||||
* Tests the Native Azure file system (WASB) against an actual blob store if
|
||||
* provided in the environment.
|
||||
*/
|
||||
public class TestNativeAzureFileSystemLive extends
|
||||
NativeAzureFileSystemBaseTest {
|
||||
|
||||
@Override
|
||||
protected AzureBlobStorageTestAccount createTestAccount() throws Exception {
|
||||
return AzureBlobStorageTestAccount.create();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,28 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
public class TestNativeAzureFileSystemMocked extends
|
||||
NativeAzureFileSystemBaseTest {
|
||||
|
||||
@Override
|
||||
protected AzureBlobStorageTestAccount createTestAccount() throws Exception {
|
||||
return AzureBlobStorageTestAccount.createMock();
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,45 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assume.assumeTrue;
|
||||
|
||||
import org.apache.hadoop.fs.FSMainOperationsBaseTest;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
|
||||
public class TestNativeAzureFileSystemOperationsMocked extends
|
||||
FSMainOperationsBaseTest {
|
||||
|
||||
public TestNativeAzureFileSystemOperationsMocked() {
|
||||
super("/tmp/TestNativeAzureFileSystemOperationsMocked");
|
||||
}
|
||||
|
||||
@Override
|
||||
protected FileSystem createFileSystem() throws Exception {
|
||||
return AzureBlobStorageTestAccount.createMock().getFileSystem();
|
||||
}
|
||||
|
||||
public void testListStatusThrowsExceptionForUnreadableDir() throws Exception {
|
||||
System.out
|
||||
.println("Skipping testListStatusThrowsExceptionForUnreadableDir since WASB"
|
||||
+ " doesn't honor directory permissions.");
|
||||
assumeTrue(!Path.WINDOWS);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,169 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
|
||||
import java.util.HashMap;
|
||||
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.permission.FsPermission;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
/**
|
||||
* Tests that WASB handles things gracefully when users add blobs to the Azure
|
||||
* Storage container from outside WASB's control.
|
||||
*/
|
||||
public class TestOutOfBandAzureBlobOperations {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
private FileSystem fs;
|
||||
private InMemoryBlockBlobStore backingStore;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
fs = testAccount.getFileSystem();
|
||||
backingStore = testAccount.getMockStorage().getBackingStore();
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
testAccount.cleanup();
|
||||
fs = null;
|
||||
backingStore = null;
|
||||
}
|
||||
|
||||
private void createEmptyBlobOutOfBand(String path) {
|
||||
backingStore.setContent(AzureBlobStorageTestAccount.toMockUri(path),
|
||||
new byte[] { 1, 2 }, new HashMap<String, String>());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testImplicitFolderListed() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
|
||||
// List the blob itself.
|
||||
FileStatus[] obtained = fs.listStatus(new Path("/root/b"));
|
||||
assertNotNull(obtained);
|
||||
assertEquals(1, obtained.length);
|
||||
assertFalse(obtained[0].isDirectory());
|
||||
assertEquals("/root/b", obtained[0].getPath().toUri().getPath());
|
||||
|
||||
// List the directory
|
||||
obtained = fs.listStatus(new Path("/root"));
|
||||
assertNotNull(obtained);
|
||||
assertEquals(1, obtained.length);
|
||||
assertFalse(obtained[0].isDirectory());
|
||||
assertEquals("/root/b", obtained[0].getPath().toUri().getPath());
|
||||
|
||||
// Get the directory's file status
|
||||
FileStatus dirStatus = fs.getFileStatus(new Path("/root"));
|
||||
assertNotNull(dirStatus);
|
||||
assertTrue(dirStatus.isDirectory());
|
||||
assertEquals("/root", dirStatus.getPath().toUri().getPath());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testImplicitFolderDeleted() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
assertTrue(fs.exists(new Path("/root")));
|
||||
assertTrue(fs.delete(new Path("/root"), true));
|
||||
assertFalse(fs.exists(new Path("/root")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFileInImplicitFolderDeleted() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
assertTrue(fs.exists(new Path("/root")));
|
||||
assertTrue(fs.delete(new Path("/root/b"), true));
|
||||
assertTrue(fs.exists(new Path("/root")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testFileAndImplicitFolderSameName() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
createEmptyBlobOutOfBand("root/b/c");
|
||||
FileStatus[] listResult = fs.listStatus(new Path("/root/b"));
|
||||
// File should win.
|
||||
assertEquals(1, listResult.length);
|
||||
assertFalse(listResult[0].isDirectory());
|
||||
try {
|
||||
// Trying to delete root/b/c would cause a dilemma for WASB, so
|
||||
// it should throw.
|
||||
fs.delete(new Path("/root/b/c"), true);
|
||||
assertTrue("Should've thrown.", false);
|
||||
} catch (AzureException e) {
|
||||
assertEquals("File /root/b/c has a parent directory /root/b"
|
||||
+ " which is also a file. Can't resolve.", e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private static enum DeepCreateTestVariation {
|
||||
File, Folder
|
||||
};
|
||||
|
||||
/**
|
||||
* Tests that when we create the file (or folder) x/y/z, we also create
|
||||
* explicit folder blobs for x and x/y
|
||||
*/
|
||||
@Test
|
||||
public void testCreatingDeepFileCreatesExplicitFolder() throws Exception {
|
||||
for (DeepCreateTestVariation variation : DeepCreateTestVariation.values()) {
|
||||
switch (variation) {
|
||||
case File:
|
||||
assertTrue(fs.createNewFile(new Path("/x/y/z")));
|
||||
break;
|
||||
case Folder:
|
||||
assertTrue(fs.mkdirs(new Path("/x/y/z")));
|
||||
break;
|
||||
}
|
||||
assertTrue(backingStore
|
||||
.exists(AzureBlobStorageTestAccount.toMockUri("x")));
|
||||
assertTrue(backingStore.exists(AzureBlobStorageTestAccount
|
||||
.toMockUri("x/y")));
|
||||
fs.delete(new Path("/x"), true);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetPermissionOnImplicitFolder() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
FsPermission newPermission = new FsPermission((short) 0600);
|
||||
fs.setPermission(new Path("/root"), newPermission);
|
||||
FileStatus newStatus = fs.getFileStatus(new Path("/root"));
|
||||
assertNotNull(newStatus);
|
||||
assertEquals(newPermission, newStatus.getPermission());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSetOwnerOnImplicitFolder() throws Exception {
|
||||
createEmptyBlobOutOfBand("root/b");
|
||||
fs.setOwner(new Path("/root"), "newOwner", null);
|
||||
FileStatus newStatus = fs.getFileStatus(new Path("/root"));
|
||||
assertNotNull(newStatus);
|
||||
assertEquals("newOwner", newStatus.getOwner());
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,182 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import org.apache.hadoop.fs.FSDataOutputStream;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.security.UserGroupInformation;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.microsoft.windowsazure.storage.blob.BlobOutputStream;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
|
||||
public class TestOutOfBandAzureBlobOperationsLive {
|
||||
private FileSystem fs;
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.create();
|
||||
if (testAccount != null) {
|
||||
fs = testAccount.getFileSystem();
|
||||
}
|
||||
assumeNotNull(testAccount);
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
fs = null;
|
||||
}
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
// creating a file out-of-band would confuse mkdirs("<oobfilesUncleFolder>")
|
||||
// eg oob creation of "user/<name>/testFolder/a/input/file"
|
||||
// Then wasb creation of "user/<name>/testFolder/a/output" fails
|
||||
@Test
|
||||
public void outOfBandFolder_uncleMkdirs() throws Exception {
|
||||
|
||||
// NOTE: manual use of CloubBlockBlob targets working directory explicitly.
|
||||
// WASB driver methods prepend working directory implicitly.
|
||||
String workingDir = "user/"
|
||||
+ UserGroupInformation.getCurrentUser().getShortUserName() + "/";
|
||||
|
||||
CloudBlockBlob blob = testAccount.getBlobReference(workingDir
|
||||
+ "testFolder1/a/input/file");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
assertTrue(fs.exists(new Path("testFolder1/a/input/file")));
|
||||
|
||||
Path targetFolder = new Path("testFolder1/a/output");
|
||||
assertTrue(fs.mkdirs(targetFolder));
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
@Test
|
||||
public void outOfBandFolder_parentDelete() throws Exception {
|
||||
|
||||
// NOTE: manual use of CloubBlockBlob targets working directory explicitly.
|
||||
// WASB driver methods prepend working directory implicitly.
|
||||
String workingDir = "user/"
|
||||
+ UserGroupInformation.getCurrentUser().getShortUserName() + "/";
|
||||
CloudBlockBlob blob = testAccount.getBlobReference(workingDir
|
||||
+ "testFolder2/a/input/file");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
assertTrue(fs.exists(new Path("testFolder2/a/input/file")));
|
||||
|
||||
Path targetFolder = new Path("testFolder2/a/input");
|
||||
assertTrue(fs.delete(targetFolder, true));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void outOfBandFolder_rootFileDelete() throws Exception {
|
||||
|
||||
CloudBlockBlob blob = testAccount.getBlobReference("fileY");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
assertTrue(fs.exists(new Path("/fileY")));
|
||||
assertTrue(fs.delete(new Path("/fileY"), true));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void outOfBandFolder_firstLevelFolderDelete() throws Exception {
|
||||
|
||||
CloudBlockBlob blob = testAccount.getBlobReference("folderW/file");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
assertTrue(fs.exists(new Path("/folderW")));
|
||||
assertTrue(fs.exists(new Path("/folderW/file")));
|
||||
assertTrue(fs.delete(new Path("/folderW"), true));
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
@Test
|
||||
public void outOfBandFolder_siblingCreate() throws Exception {
|
||||
|
||||
// NOTE: manual use of CloubBlockBlob targets working directory explicitly.
|
||||
// WASB driver methods prepend working directory implicitly.
|
||||
String workingDir = "user/"
|
||||
+ UserGroupInformation.getCurrentUser().getShortUserName() + "/";
|
||||
CloudBlockBlob blob = testAccount.getBlobReference(workingDir
|
||||
+ "testFolder3/a/input/file");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
assertTrue(fs.exists(new Path("testFolder3/a/input/file")));
|
||||
|
||||
Path targetFile = new Path("testFolder3/a/input/file2");
|
||||
FSDataOutputStream s2 = fs.create(targetFile);
|
||||
s2.close();
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
// creating a new file in the root folder
|
||||
@Test
|
||||
public void outOfBandFolder_create_rootDir() throws Exception {
|
||||
Path targetFile = new Path("/newInRoot");
|
||||
FSDataOutputStream s2 = fs.create(targetFile);
|
||||
s2.close();
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
@Test
|
||||
public void outOfBandFolder_rename() throws Exception {
|
||||
|
||||
// NOTE: manual use of CloubBlockBlob targets working directory explicitly.
|
||||
// WASB driver methods prepend working directory implicitly.
|
||||
String workingDir = "user/"
|
||||
+ UserGroupInformation.getCurrentUser().getShortUserName() + "/";
|
||||
CloudBlockBlob blob = testAccount.getBlobReference(workingDir
|
||||
+ "testFolder4/a/input/file");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
|
||||
Path srcFilePath = new Path("testFolder4/a/input/file");
|
||||
assertTrue(fs.exists(srcFilePath));
|
||||
|
||||
Path destFilePath = new Path("testFolder4/a/input/file2");
|
||||
fs.rename(srcFilePath, destFilePath);
|
||||
}
|
||||
|
||||
// scenario for this particular test described at MONARCH-HADOOP-764
|
||||
@Test
|
||||
public void outOfBandFolder_rename_rootLevelFiles() throws Exception {
|
||||
|
||||
// NOTE: manual use of CloubBlockBlob targets working directory explicitly.
|
||||
// WASB driver methods prepend working directory implicitly.
|
||||
CloudBlockBlob blob = testAccount.getBlobReference("fileX");
|
||||
BlobOutputStream s = blob.openOutputStream();
|
||||
s.close();
|
||||
|
||||
Path srcFilePath = new Path("/fileX");
|
||||
assertTrue(fs.exists(srcFilePath));
|
||||
|
||||
Path destFilePath = new Path("/fileXrename");
|
||||
fs.rename(srcFilePath, destFilePath);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,82 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
|
||||
import java.io.File;
|
||||
|
||||
import org.apache.commons.io.FileUtils;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.util.Shell;
|
||||
import org.junit.Assert;
|
||||
import org.junit.Test;
|
||||
|
||||
public class TestShellDecryptionKeyProvider {
|
||||
public static final Log LOG = LogFactory
|
||||
.getLog(TestShellDecryptionKeyProvider.class);
|
||||
private static File TEST_ROOT_DIR = new File(System.getProperty(
|
||||
"test.build.data", "/tmp"), "TestShellDecryptionKeyProvider");
|
||||
|
||||
@Test
|
||||
public void testScriptPathNotSpecified() throws Exception {
|
||||
if (!Shell.WINDOWS) {
|
||||
return;
|
||||
}
|
||||
ShellDecryptionKeyProvider provider = new ShellDecryptionKeyProvider();
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
String key = "key";
|
||||
|
||||
conf.set(SimpleKeyProvider.KEY_ACCOUNT_KEY_PREFIX + account, key);
|
||||
try {
|
||||
provider.getStorageAccountKey(account, conf);
|
||||
Assert
|
||||
.fail("fs.azure.shellkeyprovider.script is not specified, we should throw");
|
||||
} catch (KeyProviderException e) {
|
||||
LOG.info("Received an expected exception: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testValidScript() throws Exception {
|
||||
if (!Shell.WINDOWS) {
|
||||
return;
|
||||
}
|
||||
String expectedResult = "decretedKey";
|
||||
|
||||
// Create a simple script which echoes the given key plus the given
|
||||
// expected result (so that we validate both script input and output)
|
||||
File scriptFile = new File(TEST_ROOT_DIR, "testScript.cmd");
|
||||
FileUtils.writeStringToFile(scriptFile, "@echo %1 " + expectedResult);
|
||||
|
||||
ShellDecryptionKeyProvider provider = new ShellDecryptionKeyProvider();
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
String key = "key1";
|
||||
conf.set(SimpleKeyProvider.KEY_ACCOUNT_KEY_PREFIX + account, key);
|
||||
conf.set(ShellDecryptionKeyProvider.KEY_ACCOUNT_SHELLKEYPROVIDER_SCRIPT,
|
||||
"cmd /c " + scriptFile.getAbsolutePath());
|
||||
|
||||
String result = provider.getStorageAccountKey(account, conf);
|
||||
assertEquals(key + " " + expectedResult, result);
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.FSDataOutputStream;
|
||||
import org.apache.hadoop.fs.FileStatus;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
import org.junit.Test;
|
||||
|
||||
public class TestWasbFsck {
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
private FileSystem fs;
|
||||
private InMemoryBlockBlobStore backingStore;
|
||||
|
||||
@Before
|
||||
public void setUp() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
fs = testAccount.getFileSystem();
|
||||
backingStore = testAccount.getMockStorage().getBackingStore();
|
||||
}
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
testAccount.cleanup();
|
||||
fs = null;
|
||||
backingStore = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts the number of temporary blobs in the backing store.
|
||||
*/
|
||||
private int getNumTempBlobs() {
|
||||
int count = 0;
|
||||
for (String key : backingStore.getKeys()) {
|
||||
if (key.contains(NativeAzureFileSystem.AZURE_TEMP_FOLDER)) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
private void runFsck(String command) throws Exception {
|
||||
Configuration conf = fs.getConf();
|
||||
// Set the dangling cutoff to zero, so every temp blob is considered
|
||||
// dangling.
|
||||
conf.setInt(NativeAzureFileSystem.AZURE_TEMP_EXPIRY_PROPERTY_NAME, 0);
|
||||
WasbFsck fsck = new WasbFsck(conf);
|
||||
fsck.setMockFileSystemForTesting(fs);
|
||||
fsck.run(new String[] { AzureBlobStorageTestAccount.MOCK_WASB_URI, command });
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that we delete dangling files properly
|
||||
*/
|
||||
@Test
|
||||
public void testDelete() throws Exception {
|
||||
Path danglingFile = new Path("/crashedInTheMiddle");
|
||||
|
||||
// Create a file and leave it dangling and try to delete it.
|
||||
FSDataOutputStream stream = fs.create(danglingFile);
|
||||
stream.write(new byte[] { 1, 2, 3 });
|
||||
stream.flush();
|
||||
|
||||
// Now we should still only see a zero-byte file in this place
|
||||
FileStatus fileStatus = fs.getFileStatus(danglingFile);
|
||||
assertNotNull(fileStatus);
|
||||
assertEquals(0, fileStatus.getLen());
|
||||
assertEquals(1, getNumTempBlobs());
|
||||
|
||||
// Run WasbFsck -delete to delete the file.
|
||||
runFsck("-delete");
|
||||
|
||||
// Now we should see no trace of the file.
|
||||
assertEquals(0, getNumTempBlobs());
|
||||
assertFalse(fs.exists(danglingFile));
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,392 @@
|
||||
/**
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.hadoop.fs.azure;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assert.fail;
|
||||
import static org.junit.Assume.assumeNotNull;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.util.Date;
|
||||
import java.util.EnumSet;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.fs.FileSystem;
|
||||
import org.apache.hadoop.fs.Path;
|
||||
import org.apache.hadoop.fs.azure.AzureBlobStorageTestAccount.CreateOptions;
|
||||
import org.junit.After;
|
||||
import org.junit.Assert;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlobContainer;
|
||||
import com.microsoft.windowsazure.storage.blob.CloudBlockBlob;
|
||||
|
||||
public class TestWasbUriAndConfiguration {
|
||||
|
||||
private static final int FILE_SIZE = 4096;
|
||||
private static final String PATH_DELIMITER = "/";
|
||||
|
||||
protected String accountName;
|
||||
protected String accountKey;
|
||||
protected static Configuration conf = null;
|
||||
|
||||
private AzureBlobStorageTestAccount testAccount;
|
||||
|
||||
@After
|
||||
public void tearDown() throws Exception {
|
||||
if (testAccount != null) {
|
||||
testAccount.cleanup();
|
||||
testAccount = null;
|
||||
}
|
||||
}
|
||||
|
||||
private boolean validateIOStreams(Path filePath) throws IOException {
|
||||
// Capture the file system from the test account.
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
return validateIOStreams(fs, filePath);
|
||||
}
|
||||
|
||||
private boolean validateIOStreams(FileSystem fs, Path filePath)
|
||||
throws IOException {
|
||||
|
||||
// Create and write a file
|
||||
OutputStream outputStream = fs.create(filePath);
|
||||
outputStream.write(new byte[FILE_SIZE]);
|
||||
outputStream.close();
|
||||
|
||||
// Return true if the the count is equivalent to the file size.
|
||||
return (FILE_SIZE == readInputStream(fs, filePath));
|
||||
}
|
||||
|
||||
private int readInputStream(Path filePath) throws IOException {
|
||||
// Capture the file system from the test account.
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
return readInputStream(fs, filePath);
|
||||
}
|
||||
|
||||
private int readInputStream(FileSystem fs, Path filePath) throws IOException {
|
||||
// Read the file
|
||||
InputStream inputStream = fs.open(filePath);
|
||||
int count = 0;
|
||||
while (inputStream.read() >= 0) {
|
||||
count++;
|
||||
}
|
||||
inputStream.close();
|
||||
|
||||
// Return true if the the count is equivalent to the file size.
|
||||
return count;
|
||||
}
|
||||
|
||||
// Positive tests to exercise making a connection with to Azure account using
|
||||
// account key.
|
||||
@Test
|
||||
public void testConnectUsingKey() throws Exception {
|
||||
|
||||
testAccount = AzureBlobStorageTestAccount.create();
|
||||
assumeNotNull(testAccount);
|
||||
|
||||
// Validate input and output on the connection.
|
||||
assertTrue(validateIOStreams(new Path("/wasb_scheme")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testConnectUsingSAS() throws Exception {
|
||||
// Create the test account with SAS credentials.
|
||||
testAccount = AzureBlobStorageTestAccount.create("",
|
||||
EnumSet.of(CreateOptions.UseSas, CreateOptions.CreateContainer));
|
||||
assumeNotNull(testAccount);
|
||||
// Validate input and output on the connection.
|
||||
// NOTE: As of 4/15/2013, Azure Storage has a deficiency that prevents the
|
||||
// full scenario from working (CopyFromBlob doesn't work with SAS), so
|
||||
// just do a minor check until that is corrected.
|
||||
assertFalse(testAccount.getFileSystem().exists(new Path("/IDontExist")));
|
||||
//assertTrue(validateIOStreams(new Path("/sastest.txt")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testConnectUsingSASReadonly() throws Exception {
|
||||
// Create the test account with SAS credentials.
|
||||
testAccount = AzureBlobStorageTestAccount.create("", EnumSet.of(
|
||||
CreateOptions.UseSas, CreateOptions.CreateContainer,
|
||||
CreateOptions.Readonly));
|
||||
assumeNotNull(testAccount);
|
||||
|
||||
// Create a blob in there
|
||||
final String blobKey = "blobForReadonly";
|
||||
CloudBlobContainer container = testAccount.getRealContainer();
|
||||
CloudBlockBlob blob = container.getBlockBlobReference(blobKey);
|
||||
ByteArrayInputStream inputStream = new ByteArrayInputStream(new byte[] { 1,
|
||||
2, 3 });
|
||||
blob.upload(inputStream, 3);
|
||||
inputStream.close();
|
||||
|
||||
// Make sure we can read it from the file system
|
||||
Path filePath = new Path("/" + blobKey);
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
assertTrue(fs.exists(filePath));
|
||||
byte[] obtained = new byte[3];
|
||||
DataInputStream obtainedInputStream = fs.open(filePath);
|
||||
obtainedInputStream.readFully(obtained);
|
||||
obtainedInputStream.close();
|
||||
assertEquals(3, obtained[2]);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testConnectUsingAnonymous() throws Exception {
|
||||
|
||||
// Create test account with anonymous credentials
|
||||
testAccount = AzureBlobStorageTestAccount.createAnonymous("testWasb.txt",
|
||||
FILE_SIZE);
|
||||
assumeNotNull(testAccount);
|
||||
|
||||
// Read the file from the public folder using anonymous credentials.
|
||||
assertEquals(FILE_SIZE, readInputStream(new Path("/testWasb.txt")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testConnectToEmulator() throws Exception {
|
||||
testAccount = AzureBlobStorageTestAccount.createForEmulator();
|
||||
assumeNotNull(testAccount);
|
||||
assertTrue(validateIOStreams(new Path("/testFile")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests that we can connect to fully qualified accounts outside of
|
||||
* blob.core.windows.net
|
||||
*/
|
||||
@Test
|
||||
public void testConnectToFullyQualifiedAccountMock() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
AzureBlobStorageTestAccount.setMockAccountKey(conf,
|
||||
"mockAccount.mock.authority.net");
|
||||
AzureNativeFileSystemStore store = new AzureNativeFileSystemStore();
|
||||
MockStorageInterface mockStorage = new MockStorageInterface();
|
||||
store.setAzureStorageInteractionLayer(mockStorage);
|
||||
NativeAzureFileSystem fs = new NativeAzureFileSystem(store);
|
||||
fs.initialize(
|
||||
new URI("wasb://mockContainer@mockAccount.mock.authority.net"), conf);
|
||||
fs.createNewFile(new Path("/x"));
|
||||
assertTrue(mockStorage.getBackingStore().exists(
|
||||
"http://mockAccount.mock.authority.net/mockContainer/x"));
|
||||
fs.close();
|
||||
}
|
||||
|
||||
public void testConnectToRoot() throws Exception {
|
||||
|
||||
// Set up blob names.
|
||||
final String blobPrefix = String.format("wasbtests-%s-%tQ-blob",
|
||||
System.getProperty("user.name"), new Date());
|
||||
final String inblobName = blobPrefix + "_In" + ".txt";
|
||||
final String outblobName = blobPrefix + "_Out" + ".txt";
|
||||
|
||||
// Create test account with default root access.
|
||||
testAccount = AzureBlobStorageTestAccount.createRoot(inblobName, FILE_SIZE);
|
||||
assumeNotNull(testAccount);
|
||||
|
||||
// Read the file from the default container.
|
||||
assertEquals(FILE_SIZE, readInputStream(new Path(PATH_DELIMITER
|
||||
+ inblobName)));
|
||||
|
||||
try {
|
||||
// Capture file system.
|
||||
FileSystem fs = testAccount.getFileSystem();
|
||||
|
||||
// Create output path and open an output stream to the root folder.
|
||||
Path outputPath = new Path(PATH_DELIMITER + outblobName);
|
||||
OutputStream outputStream = fs.create(outputPath);
|
||||
fail("Expected an AzureException when writing to root folder.");
|
||||
outputStream.write(new byte[FILE_SIZE]);
|
||||
outputStream.close();
|
||||
} catch (AzureException e) {
|
||||
assertTrue(true);
|
||||
} catch (Exception e) {
|
||||
String errMsg = String.format(
|
||||
"Expected AzureException but got %s instead.", e);
|
||||
assertTrue(errMsg, false);
|
||||
}
|
||||
}
|
||||
|
||||
// Positive tests to exercise throttling I/O path. Connections are made to an
|
||||
// Azure account using account key.
|
||||
//
|
||||
public void testConnectWithThrottling() throws Exception {
|
||||
|
||||
testAccount = AzureBlobStorageTestAccount.createThrottled();
|
||||
|
||||
// Validate input and output on the connection.
|
||||
assertTrue(validateIOStreams(new Path("/wasb_scheme")));
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a file and writes a single byte with the given value in it.
|
||||
*/
|
||||
private static void writeSingleByte(FileSystem fs, Path testFile, int toWrite)
|
||||
throws Exception {
|
||||
OutputStream outputStream = fs.create(testFile);
|
||||
outputStream.write(toWrite);
|
||||
outputStream.close();
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the file given and makes sure that it's a single-byte file with the
|
||||
* given value in it.
|
||||
*/
|
||||
private static void assertSingleByteValue(FileSystem fs, Path testFile,
|
||||
int expectedValue) throws Exception {
|
||||
InputStream inputStream = fs.open(testFile);
|
||||
int byteRead = inputStream.read();
|
||||
assertTrue("File unexpectedly empty: " + testFile, byteRead >= 0);
|
||||
assertTrue("File has more than a single byte: " + testFile,
|
||||
inputStream.read() < 0);
|
||||
inputStream.close();
|
||||
assertEquals("Unxpected content in: " + testFile, expectedValue, byteRead);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMultipleContainers() throws Exception {
|
||||
AzureBlobStorageTestAccount firstAccount = AzureBlobStorageTestAccount
|
||||
.create("first"), secondAccount = AzureBlobStorageTestAccount
|
||||
.create("second");
|
||||
assumeNotNull(firstAccount);
|
||||
assumeNotNull(secondAccount);
|
||||
try {
|
||||
FileSystem firstFs = firstAccount.getFileSystem(), secondFs = secondAccount
|
||||
.getFileSystem();
|
||||
Path testFile = new Path("/testWasb");
|
||||
assertTrue(validateIOStreams(firstFs, testFile));
|
||||
assertTrue(validateIOStreams(secondFs, testFile));
|
||||
// Make sure that we're really dealing with two file systems here.
|
||||
writeSingleByte(firstFs, testFile, 5);
|
||||
writeSingleByte(secondFs, testFile, 7);
|
||||
assertSingleByteValue(firstFs, testFile, 5);
|
||||
assertSingleByteValue(secondFs, testFile, 7);
|
||||
} finally {
|
||||
firstAccount.cleanup();
|
||||
secondAccount.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testDefaultKeyProvider() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
String key = "testkey";
|
||||
|
||||
conf.set(SimpleKeyProvider.KEY_ACCOUNT_KEY_PREFIX + account, key);
|
||||
|
||||
String result = AzureNativeFileSystemStore.getAccountKeyFromConfiguration(
|
||||
account, conf);
|
||||
assertEquals(key, result);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testValidKeyProvider() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
String key = "testkey";
|
||||
|
||||
conf.set(SimpleKeyProvider.KEY_ACCOUNT_KEY_PREFIX + account, key);
|
||||
conf.setClass("fs.azure.account.keyprovider." + account,
|
||||
SimpleKeyProvider.class, KeyProvider.class);
|
||||
String result = AzureNativeFileSystemStore.getAccountKeyFromConfiguration(
|
||||
account, conf);
|
||||
assertEquals(key, result);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testInvalidKeyProviderNonexistantClass() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
|
||||
conf.set("fs.azure.account.keyprovider." + account,
|
||||
"org.apache.Nonexistant.Class");
|
||||
try {
|
||||
AzureNativeFileSystemStore.getAccountKeyFromConfiguration(account, conf);
|
||||
Assert.fail("Nonexistant key provider class should have thrown a "
|
||||
+ "KeyProviderException");
|
||||
} catch (KeyProviderException e) {
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testInvalidKeyProviderWrongClass() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
String account = "testacct";
|
||||
|
||||
conf.set("fs.azure.account.keyprovider." + account, "java.lang.String");
|
||||
try {
|
||||
AzureNativeFileSystemStore.getAccountKeyFromConfiguration(account, conf);
|
||||
Assert.fail("Key provider class that doesn't implement KeyProvider "
|
||||
+ "should have thrown a KeyProviderException");
|
||||
} catch (KeyProviderException e) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tests the cases when the URI is specified with no authority, i.e.
|
||||
* wasb:///path/to/file.
|
||||
*/
|
||||
@Test
|
||||
public void testNoUriAuthority() throws Exception {
|
||||
// For any combination of default FS being asv(s)/wasb(s)://c@a/ and
|
||||
// the actual URI being asv(s)/wasb(s):///, it should work.
|
||||
|
||||
String[] wasbAliases = new String[] { "wasb", "wasbs" };
|
||||
for (String defaultScheme : wasbAliases){
|
||||
for (String wantedScheme : wasbAliases) {
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
Configuration conf = testAccount.getFileSystem().getConf();
|
||||
String authority = testAccount.getFileSystem().getUri().getAuthority();
|
||||
URI defaultUri = new URI(defaultScheme, authority, null, null, null);
|
||||
conf.set("fs.default.name", defaultUri.toString());
|
||||
URI wantedUri = new URI(wantedScheme + ":///random/path");
|
||||
NativeAzureFileSystem obtained = (NativeAzureFileSystem) FileSystem
|
||||
.get(wantedUri, conf);
|
||||
assertNotNull(obtained);
|
||||
assertEquals(new URI(wantedScheme, authority, null, null, null),
|
||||
obtained.getUri());
|
||||
// Make sure makeQualified works as expected
|
||||
Path qualified = obtained.makeQualified(new Path(wantedUri));
|
||||
assertEquals(new URI(wantedScheme, authority, wantedUri.getPath(),
|
||||
null, null), qualified.toUri());
|
||||
// Cleanup for the next iteration to not cache anything in FS
|
||||
testAccount.cleanup();
|
||||
FileSystem.closeAll();
|
||||
}
|
||||
}
|
||||
// If the default FS is not a WASB FS, then specifying a URI without
|
||||
// authority for the Azure file system should throw.
|
||||
testAccount = AzureBlobStorageTestAccount.createMock();
|
||||
Configuration conf = testAccount.getFileSystem().getConf();
|
||||
conf.set("fs.default.name", "file:///");
|
||||
try {
|
||||
FileSystem.get(new URI("wasb:///random/path"), conf);
|
||||
fail("Should've thrown.");
|
||||
} catch (IllegalArgumentException e) {
|
||||
}
|
||||
}
|
||||
}
|
||||
49
hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
Normal file
49
hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
Normal file
@ -0,0 +1,49 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
-->
|
||||
|
||||
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
|
||||
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
|
||||
|
||||
<property>
|
||||
<name>fs.wasb.impl</name>
|
||||
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<name>fs.wasbs.impl</name>
|
||||
<value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value>
|
||||
</property>
|
||||
|
||||
|
||||
<!-- For tests against live azure, provide the following account information -->
|
||||
<!--
|
||||
<property>
|
||||
<name>fs.azure.test.account.name</name>
|
||||
<value>{ACCOUNTNAME}.blob.core.windows.net</value>
|
||||
</property>
|
||||
<property>
|
||||
<name>fs.azure.account.key.{ACCOUNTNAME}.blob.core.windows.net</name>
|
||||
<value>{ACCOUNTKEY}</value>
|
||||
</property>
|
||||
-->
|
||||
|
||||
<!-- For tests against azure-emulator -->
|
||||
<!--
|
||||
<property>
|
||||
<name>fs.azure.test.emulator</name>
|
||||
<value>true</value>
|
||||
</property>
|
||||
-->
|
||||
</configuration>
|
||||
@ -0,0 +1,23 @@
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
# contributor license agreements. See the NOTICE file distributed with
|
||||
# this work for additional information regarding copyright ownership.
|
||||
# The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
# (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# log4j configuration used during build and unit tests
|
||||
|
||||
log4j.rootLogger=INFO,stdout
|
||||
log4j.threshhold=ALL
|
||||
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
|
||||
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
|
||||
@ -83,6 +83,12 @@
|
||||
<scope>compile</scope>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-azure</artifactId>
|
||||
<scope>compile</scope>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-sls</artifactId>
|
||||
|
||||
@ -42,6 +42,7 @@
|
||||
<module>hadoop-pipes</module>
|
||||
<module>hadoop-openstack</module>
|
||||
<module>hadoop-sls</module>
|
||||
<module>hadoop-azure</module>
|
||||
</modules>
|
||||
|
||||
<build>
|
||||
|
||||
Loading…
Reference in New Issue
Block a user