big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran

Browse Source
This commit is contained in:
Steve Loughran 2017-02-24 13:24:59 +00:00
parent 9c22a91662
commit e60c6543d5
No known key found for this signature in database
GPG Key ID: 950CC3E032B79CA2
1 changed files with 122 additions and 111 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

233
hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
View File

@ -20,20 +20,20 @@
* [Usage](#Usage) * [Usage](#Usage)
* [Concepts](#Concepts) * [Concepts](#Concepts)
* [OAuth2 Support](#OAuth2_Support) * [OAuth2 Support](#OAuth2_Support)
* [Configuring Credentials & FileSystem](#Configuring_Credentials) * [Configuring Credentials and FileSystem](#Configuring_Credentials)
* [Using Refresh Token](#Refresh_Token) * [Using Refresh Token](#Refresh_Token)
* [Using Client Keys](#Client_Credential_Token) * [Using Client Keys](#Client_Credential_Token)
* [Protecting the Credentials with Credential Providers](#Credential_Provider) * [Protecting the Credentials with Credential Providers](#Credential_Provider)
* [Enabling ADL Filesystem](#Enabling_ADL) * [Enabling ADL Filesystem](#Enabling_ADL)
* [Accessing adl URLs](#Accessing_adl_URLs) * [Accessing `adl` URLs](#Accessing_adl_URLs)
* [User/Group Representation](#OIDtoUPNConfiguration) * [User/Group Representation](#OIDtoUPNConfiguration)
* [Testing the hadoop-azure Module](#Testing_the_hadoop-azure_Module) * [Testing the `hadoop-azure` Module](#Testing_the_hadoop-azure_Module)
## <a name="Introduction" />Introduction ## <a name="Introduction" />Introduction
The hadoop-azure-datalake module provides support for integration with The `hadoop-azure-datalake` module provides support for integration with the
[Azure Data Lake Store]( https://azure.microsoft.com/en-in/documentation/services/data-lake-store/). [Azure Data Lake Store](https://azure.microsoft.com/en-in/documentation/services/data-lake-store/).
The jar file is named azure-datalake-store.jar. This support comes via the JAR file `azure-datalake-store.jar`.
## <a name="Features" />Features ## <a name="Features" />Features
@ -43,13 +43,14 @@ The jar file is named azure-datalake-store.jar.
* Can act as a source of data in a MapReduce job, or a sink. * Can act as a source of data in a MapReduce job, or a sink.
* Tested on both Linux and Windows. * Tested on both Linux and Windows.
* Tested for scale. * Tested for scale.
* API setOwner/setAcl/removeAclEntries/modifyAclEntries accepts UPN or OID * API `setOwner()`, `setAcl`, `removeAclEntries()`, `modifyAclEntries()` accepts UPN or OID
(Object ID) as user and group name. (Object ID) as user and group names.
## <a name="Limitations" />Limitations ## <a name="Limitations" />Limitations
Partial or no support for the following operations : Partial or no support for the following operations :
* Operation on Symbolic Link * Operation on Symbolic Links
* Proxy Users * Proxy Users
* File Truncate * File Truncate
* File Checksum * File Checksum
@ -58,55 +59,71 @@ Partial or no support for the following operations :
* Extended Attributes(XAttrs) Operations * Extended Attributes(XAttrs) Operations
* Snapshot Operations * Snapshot Operations
* Delegation Token Operations * Delegation Token Operations
* User and group information returned as ListStatus and GetFileStatus is in form of GUID associated in Azure Active Directory. * User and group information returned as `listStatus()` and `getFileStatus()` is
in the form of the GUID associated in Azure Active Directory.
## <a name="Usage" />Usage ## <a name="Usage" />Usage
### <a name="Concepts" />Concepts ### <a name="Concepts" />Concepts
Azure Data Lake Storage access path syntax is Azure Data Lake Storage access path syntax is:
adl://<Account Name>.azuredatalakestore.net/ ```
adl://<Account Name>.azuredatalakestore.net/
```
Get started with azure data lake account with [https://azure.microsoft.com/en-in/documentation/articles/data-lake-store-get-started-portal/](https://azure.microsoft.com/en-in/documentation/articles/data-lake-store-get-started-portal/) For details on using the store, see
[**Get started with Azure Data Lake Store using the Azure Portal**](https://azure.microsoft.com/en-in/documentation/articles/data-lake-store-get-started-portal/)
#### <a name="#OAuth2_Support" />OAuth2 Support ### <a name="#OAuth2_Support" />OAuth2 Support
Usage of Azure Data Lake Storage requires OAuth2 bearer token to be present as part of the HTTPS header as per OAuth2 specification. Valid OAuth2 bearer token should be obtained from Azure Active Directory for valid users who have access to Azure Data Lake Storage Account.
Azure Active Directory (Azure AD) is Microsoft's multi-tenant cloud based directory and identity management service. See [https://azure.microsoft.com/en-in/documentation/articles/active-directory-whatis/](https://azure.microsoft.com/en-in/documentation/articles/active-directory-whatis/) Usage of Azure Data Lake Storage requires an OAuth2 bearer token to be present as
part of the HTTPS header as per the OAuth2 specification.
A valid OAuth2 bearer token must be obtained from the Azure Active Directory service
for those valid users who have access to Azure Data Lake Storage Account.
Following sections describes on OAuth2 configuration in core-site.xml. Azure Active Directory (Azure AD) is Microsoft's multi-tenant cloud based directory
and identity management service. See [*What is ActiveDirectory*](https://azure.microsoft.com/en-in/documentation/articles/active-directory-whatis/).
## <a name="Configuring_Credentials" />Configuring Credentials & FileSystem Following sections describes theOAuth2 configuration in `core-site.xml`.
Credentials can be configured using either a refresh token (associated with a user) or a client credential (analogous to a service principal).
### <a name="Refresh_Token" />Using Refresh Token #### <a name="Configuring_Credentials" />Configuring Credentials & FileSystem
Credentials can be configured using either a refresh token (associated with a user),
or a client credential (analogous to a service principal).
Add the following properties to your core-site.xml #### <a name="Refresh_Token" />Using Refresh Tokens
<property> Add the following properties to the cluster's `core-site.xml`
<name>dfs.adls.oauth2.access.token.provider.type</name>
<value>RefreshToken</value>
</property>
Application require to set Client id and OAuth2 refresh token from Azure Active Directory associated with client id. See [https://github.com/AzureAD/azure-activedirectory-library-for-java](https://github.com/AzureAD/azure-activedirectory-library-for-java). ```xml
<property>
<name>dfs.adls.oauth2.access.token.provider.type</name>
<value>RefreshToken</value>
</property>
```
Applications must set the Client id and OAuth2 refresh token from the Azure Active Directory
service associated with the client id. See [*Active Directory Library For Java*](https://github.com/AzureAD/azure-activedirectory-library-for-java).
**Do not share client id and refresh token, it must be kept secret.** **Do not share client id and refresh token, it must be kept secret.**
<property> ```xml
<name>dfs.adls.oauth2.client.id</name> <property>
<value></value> <name>dfs.adls.oauth2.client.id</name>
</property> <value></value>
</property>
<property> <property>
<name>dfs.adls.oauth2.refresh.token</name> <name>dfs.adls.oauth2.refresh.token</name>
<value></value> <value></value>
</property> </property>
```
### <a name="Client_Credential_Token" />Using Client Keys ### <a name="Client_Credential_Token" />Using Client Keys
#### Generating the Service Principal #### Generating the Service Principal
1. Go to the portal (https://portal.azure.com)
1. Go to [the portal](https://portal.azure.com)
2. Under "Browse", look for Active Directory and click on it. 2. Under "Browse", look for Active Directory and click on it.
3. Create "Web Application". Remember the name you create here - that is what you will add to your ADL account as authorized user. 3. Create "Web Application". Remember the name you create here - that is what you will add to your ADL account as authorized user.
4. Go through the wizard 4. Go through the wizard
@ -124,31 +141,31 @@ Application require to set Client id and OAuth2 refresh token from Azure Active
3. Add your user name you created in Step 6 above (note that it does not show up in the list, but will be found if you searched for the name) 3. Add your user name you created in Step 6 above (note that it does not show up in the list, but will be found if you searched for the name)
4. Add "Owner" role 4. Add "Owner" role
#### Configure core-site.xml ### Configure core-site.xml
Add the following properties to your core-site.xml Add the following properties to your `core-site.xml`
<property> ```xml
<name>dfs.adls.oauth2.refresh.url</name> <property>
<value>TOKEN ENDPOINT FROM STEP 7 ABOVE</value> <name>dfs.adls.oauth2.refresh.url</name>
</property> <value>TOKEN ENDPOINT FROM STEP 7 ABOVE</value>
</property>
<property> <property>
<name>dfs.adls.oauth2.client.id</name> <name>dfs.adls.oauth2.client.id</name>
<value>CLIENT ID FROM STEP 7 ABOVE</value> <value>CLIENT ID FROM STEP 7 ABOVE</value>
</property> </property>
<property>
<name>dfs.adls.oauth2.credential</name>
<value>PASSWORD FROM STEP 7 ABOVE</value>
</property>
<property>
<name>dfs.adls.oauth2.credential</name>
<value>PASSWORD FROM STEP 7 ABOVE</value>
</property>
```
### <a name="Credential_Provider" />Protecting the Credentials with Credential Providers ### <a name="Credential_Provider" />Protecting the Credentials with Credential Providers
In many Hadoop clusters, the core-site.xml file is world-readable. To protect In many Hadoop clusters, the `core-site.xml` file is world-readable. To protect
these credentials from prying eyes, it is recommended that you use the these credentials, it is recommended that you use the
credential provider framework to securely store them and access them through credential provider framework to securely store them and access them.
configuration.
All ADLS credential properties can be protected by credential providers. All ADLS credential properties can be protected by credential providers.
For additional reading on the credential provider API, see For additional reading on the credential provider API, see
@ -156,16 +173,16 @@ For additional reading on the credential provider API, see
#### Provisioning #### Provisioning
``` ```bash
% hadoop credential create dfs.adls.oauth2.refresh.token -value 123 hadoop credential create dfs.adls.oauth2.refresh.token -value 123
-provider localjceks://file/home/foo/adls.jceks -provider localjceks://file/home/foo/adls.jceks
% hadoop credential create dfs.adls.oauth2.credential -value 123 hadoop credential create dfs.adls.oauth2.credential -value 123
-provider localjceks://file/home/foo/adls.jceks -provider localjceks://file/home/foo/adls.jceks
``` ```
#### Configuring core-site.xml or command line property #### Configuring core-site.xml or command line property
``` ```xml
<property> <property>
<name>hadoop.security.credential.provider.path</name> <name>hadoop.security.credential.provider.path</name>
<value>localjceks://file/home/foo/adls.jceks</value> <value>localjceks://file/home/foo/adls.jceks</value>
@ -175,42 +192,28 @@ For additional reading on the credential provider API, see
#### Running DistCp #### Running DistCp
``` ```bash
% hadoop distcp hadoop distcp
[-D hadoop.security.credential.provider.path=localjceks://file/home/user/adls.jceks] [-D hadoop.security.credential.provider.path=localjceks://file/home/user/adls.jceks]
hdfs://<NameNode Hostname>:9001/user/foo/007020615 hdfs://<NameNode Hostname>:9001/user/foo/007020615
adl://<Account Name>.azuredatalakestore.net/testDir/ adl://<Account Name>.azuredatalakestore.net/testDir/
``` ```
NOTE: You may optionally add the provider path property to the distcp command NOTE: You may optionally add the provider path property to the `distcp` command
line instead of added job specific configuration to a generic core-site.xml. line instead of added job specific configuration to a generic `core-site.xml`.
The square brackets above illustrate this capability. The square brackets above illustrate this capability.`
## <a name="Enabling_ADL" />Enabling ADL Filesystem
For ADL FileSystem to take effect. Update core-site.xml with
<property>
<name>fs.adl.impl</name>
<value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.adl.impl</name>
<value>org.apache.hadoop.fs.adl.Adl</value>
</property>
### <a name="Accessing_adl_URLs" />Accessing adl URLs ### <a name="Accessing_adl_URLs" />Accessing adl URLs
After credentials are configured in core-site.xml, any Hadoop component may After credentials are configured in `core-site.xml`, any Hadoop component may
reference files in that Azure Data Lake Storage account by using URLs of the following reference files in that Azure Data Lake Storage account by using URLs of the following
format: format:
adl://<Account Name>.azuredatalakestore.net/<path> ```
adl://<Account Name>.azuredatalakestore.net/<path>
```
The schemes `adl` identify a URL on a file system backed by Azure The schemes `adl` identifies a URL on a Hadoop-compatible file system backed by Azure
Data Lake Storage. `adl` utilizes encrypted HTTPS access for all interaction with Data Lake Storage. `adl` utilizes encrypted HTTPS access for all interaction with
the Azure Data Lake Storage API. the Azure Data Lake Storage API.
@ -218,48 +221,56 @@ For example, the following
[FileSystem Shell](../hadoop-project-dist/hadoop-common/FileSystemShell.html) [FileSystem Shell](../hadoop-project-dist/hadoop-common/FileSystemShell.html)
commands demonstrate access to a storage account named `youraccount`. commands demonstrate access to a storage account named `youraccount`.
> hadoop fs -mkdir adl://yourcontainer.azuredatalakestore.net/testDir
> hadoop fs -put testFile adl://yourcontainer.azuredatalakestore.net/testDir/testFile ```bash
hadoop fs -mkdir adl://yourcontainer.azuredatalakestore.net/testDir
> hadoop fs -cat adl://yourcontainer.azuredatalakestore.net/testDir/testFile hadoop fs -put testFile adl://yourcontainer.azuredatalakestore.net/testDir/testFile
test file content
hadoop fs -cat adl://yourcontainer.azuredatalakestore.net/testDir/testFile
test file content
```
### <a name="OIDtoUPNConfiguration" />User/Group Representation ### <a name="OIDtoUPNConfiguration" />User/Group Representation
The hadoop-azure-datalake module provides support for configuring how
The `hadoop-azure-datalake` module provides support for configuring how
User/Group information is represented during User/Group information is represented during
getFileStatus/listStatus/getAclStatus. `getFileStatus()`, `listStatus()`, and `getAclStatus()` calls..
Add the following properties to your core-site.xml Add the following properties to `core-site.xml`
<property> ```xml
<name>adl.feature.ownerandgroup.enableupn</name> <property>
<value>true</value> <name>adl.feature.ownerandgroup.enableupn</name>
<description> <value>true</value>
When true : User and Group in FileStatus/AclStatus response is <description>
represented as user friendly name as per Azure AD profile. When true : User and Group in FileStatus/AclStatus response is
represented as user friendly name as per Azure AD profile.
When false (default) : User and Group in FileStatus/AclStatus When false (default) : User and Group in FileStatus/AclStatus
response is represented by the unique identifier from Azure AD response is represented by the unique identifier from Azure AD
profile (Object ID as GUID). profile (Object ID as GUID).
For performance optimization, Recommended default value.
</description>
</property>
For performance optimization, Recommended default value.
</description>
</property>
```
## <a name="Testing_the_hadoop-azure_Module" />Testing the azure-datalake-store Module ## <a name="Testing_the_hadoop-azure_Module" />Testing the azure-datalake-store Module
The hadoop-azure module includes a full suite of unit tests. Most of the tests will run without additional configuration by running mvn test. This includes tests against mocked storage, which is an in-memory emulation of Azure Data Lake Storage. The `hadoop-azure` module includes a full suite of unit tests.
Most of the tests will run without additional configuration by running mvn test.
This includes tests against mocked storage, which is an in-memory emulation of Azure Data Lake Storage.
A selection of tests can run against the Azure Data Lake Storage. To run these A selection of tests can run against the Azure Data Lake Storage. To run these
tests, please create `src/test/resources/auth-keys.xml` with Adl account tests, please create `src/test/resources/auth-keys.xml` with Adl account
information mentioned in the above sections and the following properties. information mentioned in the above sections and the following properties.
<property> ```xml
<name>dfs.adl.test.contract.enable</name> <property>
<value>true</value> <name>dfs.adl.test.contract.enable</name>
</property> <value>true</value>
</property>
<property> <property>
<name>test.fs.adl.name</name> <name>test.fs.adl.name</name>
<value>adl://yourcontainer.azuredatalakestore.net</value> <value>adl://yourcontainer.azuredatalakestore.net</value>
</property> </property>
```