From 594c70f779b277bd0b9d0a5dc98c3e9cc49b7e91 Mon Sep 17 00:00:00 2001 From: Andrew Wang Date: Thu, 7 Apr 2016 23:50:27 -0700 Subject: [PATCH] HADOOP-12951. Improve documentation on KMS ACLs and delegation tokens. Contributed by Xiao Chen. --- .../server/AuthenticationFilter.java | 63 +------------------ .../util/ZKSignerSecretProvider.java | 29 +-------- .../src/site/markdown/Configuration.md | 50 ++++++++------- .../hadoop-kms/src/site/markdown/index.md.vm | 39 ++++++++---- 4 files changed, 60 insertions(+), 121 deletions(-) diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java index 5c93fd3737..0a9b8b5b7c 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java @@ -40,85 +40,26 @@ import java.util.*; /** - *

The {@link AuthenticationFilter} enables protecting web application + * The {@link AuthenticationFilter} enables protecting web application * resources with different (pluggable) * authentication mechanisms and signer secret providers. - *

*

- * Out of the box it provides 2 authentication mechanisms: Pseudo and Kerberos SPNEGO. - *

* Additional authentication mechanisms are supported via the {@link AuthenticationHandler} interface. *

* This filter delegates to the configured authentication handler for authentication and once it obtains an * {@link AuthenticationToken} from it, sets a signed HTTP cookie with the token. For client requests * that provide the signed HTTP cookie, it verifies the validity of the cookie, extracts the user information * and lets the request proceed to the target resource. - *

- * The supported configuration properties are: - * *

* The rest of the configuration properties are specific to the {@link AuthenticationHandler} implementation and the * {@link AuthenticationFilter} will take all the properties that start with the prefix #PREFIX#, it will remove * the prefix from it and it will pass them to the the authentication handler for initialization. Properties that do * not start with the prefix will not be passed to the authentication handler initialization. - *

*

- * Out of the box it provides 3 signer secret provider implementations: - * "file", "random" and "zookeeper" - *

- * Additional signer secret providers are supported via the - * {@link SignerSecretProvider} class. - *

- * For the HTTP cookies mentioned above, the SignerSecretProvider is used to - * determine the secret to use for signing the cookies. Different - * implementations can have different behaviors. The "file" implementation - * loads the secret from a specified file. The "random" implementation uses a - * randomly generated secret that rolls over at the interval specified by the - * [#PREFIX#.]token.validity mentioned above. The "zookeeper" implementation - * is like the "random" one, except that it synchronizes the random secret - * and rollovers between multiple servers; it's meant for HA services. - *

- * The relevant configuration properties are: - * + * Details of the configurations are listed on Configuration Page *

* The "zookeeper" implementation has additional configuration properties that * must be specified; see {@link ZKSignerSecretProvider} for details. - *

- * For subclasses of AuthenticationFilter that want additional control over the - * SignerSecretProvider, they can use the following attribute set in the - * ServletContext: - * */ @InterfaceAudience.Private diff --git a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java index 5e5f0879c8..0e75cbda93 100644 --- a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java +++ b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java @@ -57,34 +57,7 @@ * {@link org.apache.hadoop.security.authentication.server.AuthenticationFilter} * for more details. *

- * The supported configuration properties are: - *

- * - * The following attribute in the ServletContext can also be set if desired: - * + * Details of the configurations are listed on Configuration Page */ @InterfaceStability.Unstable @InterfaceAudience.Private diff --git a/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md b/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md index 2f9b8606aa..2a1f73b029 100644 --- a/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md +++ b/hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md @@ -34,12 +34,11 @@ Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the S * `[PREFIX.]type`: the authentication type keyword (`simple` or \ `kerberos`) or a Authentication handler implementation. -* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set to - `string` or not specified, this is the value for the secret used to sign - the HTTP cookie. +* `[PREFIX.]signature.secret.file`: When `signer.secret.provider` is set to + `file`, this is the location of file including the secret used to sign the HTTP cookie. * `[PREFIX.]token.validity`: The validity -in seconds- of the generated - authentication token. The default value is `3600` seconds. This is also + authentication token. The default value is `36000` seconds. This is also used for the rollover interval when `signer.secret.provider` is set to `random` or `zookeeper`. @@ -50,10 +49,11 @@ Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the S authentication token. * `signer.secret.provider`: indicates the name of the SignerSecretProvider - class to use. Possible values are: `string`, `random`, - `zookeeper`, or a classname. If not specified, the `string` + class to use. Possible values are: `file`, `random`, + `zookeeper`, or a classname. If not specified, the `file` implementation will be used; and failing that, the `random` - implementation will be used. + implementation will be used. If "file" is to be used, one need to specify + `signature.secret.file` and point to the secret file. ### Kerberos Configuration @@ -232,24 +232,25 @@ The SignerSecretProvider is used to provide more advanced behaviors for the secr These are the relevant configuration properties: * `signer.secret.provider`: indicates the name of the - SignerSecretProvider class to use. Possible values are: "string", - "random", "zookeeper", or a classname. If not specified, the "string" + SignerSecretProvider class to use. Possible values are: "file", + "random", "zookeeper", or a classname. If not specified, the "file" implementation will be used; and failing that, the "random" implementation - will be used. + will be used. If "file" is to be used, one need to specify `signature.secret.file` + and point to the secret file. -* `[PREFIX.]signature.secret`: When `signer.secret.provider` is set - to `string` or not specified, this is the value for the secret used to +* `[PREFIX.]signature.secret.file`: When `signer.secret.provider` is set + to `file` or not specified, this is the value for the secret used to sign the HTTP cookie. * `[PREFIX.]token.validity`: The validity -in seconds- of the generated - authentication token. The default value is `3600` seconds. This is + authentication token. The default value is `36000` seconds. This is also used for the rollover interval when `signer.secret.provider` is set to `random` or `zookeeper`. The following configuration properties are specific to the `zookeeper` implementation: * `signer.secret.provider.zookeeper.connection.string`: Indicates the - ZooKeeper connection string to connect with. + ZooKeeper connection string to connect with. The default value is `localhost:2181` * `signer.secret.provider.zookeeper.path`: Indicates the ZooKeeper path to use for storing and retrieving the secrets. All servers @@ -266,6 +267,17 @@ The following configuration properties are specific to the `zookeeper` implement * `signer.secret.provider.zookeeper.kerberos.principal`: Set this to the Kerberos principal to use. This only required if using Kerberos. +* `signer.secret.provider.zookeeper.disconnect.on.shutdown`: Whether to close the + ZooKeeper connection when the provider is shutdown. The default value is `true`. + Only set this to `false` if a custom Curator client is being provided and + the disconnection is being handled elsewhere. + +The following attribute in the ServletContext can also be set if desired: +* `signer.secret.provider.zookeeper.curator.client`: A CuratorFramework client + object can be passed here. If given, the "zookeeper" implementation will use + this Curator client instead of creating its own, which is useful if you already + have a Curator client or want more control over its configuration. + **Example**: ```xml @@ -276,11 +288,11 @@ The following configuration properties are specific to the `zookeeper` implement signer.secret.provider - string + file - signature.secret - my_secret + signature.secret.file + /myapp/secret_file @@ -334,10 +346,6 @@ The following configuration properties are specific to the `zookeeper` implement signer.secret.provider.zookeeper.path /myapp/secrets - - signer.secret.provider.zookeeper.use.kerberos.acls - true - signer.secret.provider.zookeeper.kerberos.keytab /tmp/auth.keytab diff --git a/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm index 1472ba2a51..65854cf110 100644 --- a/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm +++ b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm @@ -284,7 +284,15 @@ The answer to "What is your first and last name?" (i.e. "CN") must be the hostna NOTE: You need to restart the KMS for the configuration changes to take effect. -$H4 KMS Access Control +$H4 ACLs (Access Control Lists) + +KMS supports ACLs (Access Control Lists) for fine-grained permission control. + +Two levels of ACLs exist in KMS: KMS ACLs and Key ACLs. KMS ACLs control access at KMS operation level, and precede Key ACLs. In particular, only if permission is granted at KMS ACLs level, shall the permission check against Key ACLs be performed. + +The configuration and usage of KMS ACLs and Key ACLs are described in the sections below. + +$H5 KMS ACLs KMS ACLs configuration are defined in the KMS `etc/hadoop/kms-acls.xml` configuration file. This file is hot-reloaded when it changes. @@ -452,7 +460,7 @@ A user accessing KMS is first checked for inclusion in the Access Control List f ``` -$H4 Key Access Control +$H5 Key ACLs KMS supports access control for all non-read operations at the Key level. All Key Access operations are classified as : @@ -466,9 +474,9 @@ These can be defined in the KMS `etc/hadoop/kms-acls.xml` as follows For all keys for which a key access has not been explicitly configured, It is possible to configure a default key access control for a subset of the operation types. -It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL is a whitelist in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL. +It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL grants access to the key, in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL. -If no ACL is configured for a specific key AND no default ACL is configured AND no root key ACL is configured for the requested operation, then access will be DENIED. +If no ACL is configured for a specific key AND no default ACL is configured AND no whitelist key ACL is configured for the requested operation, then access will be DENIED. **NOTE:** The default and whitelist key ACL does not support `ALL` operation qualifier. @@ -575,7 +583,11 @@ If no ACL is configured for a specific key AND no default ACL is configured AND $H3 KMS Delegation Token Configuration -KMS delegation token secret manager can be configured with the following properties: +KMS supports delegation tokens to authenticate to the key providers from processes without Kerberos credentials. + +KMS delegation token authentication extends the default Hadoop authentication. See [Hadoop Auth](../hadoop-auth/index.html) page for more details. + +Additionally, KMS delegation token secret manager can be configured with the following properties: ```xml @@ -590,7 +602,7 @@ KMS delegation token secret manager can be configured with the following propert hadoop.kms.authentication.delegation-token.max-lifetime.sec 604800 - Maximum lifetime of a delagation token, in seconds. Default value 7 days. + Maximum lifetime of a delegation token, in seconds. Default value 7 days. @@ -598,7 +610,7 @@ KMS delegation token secret manager can be configured with the following propert hadoop.kms.authentication.delegation-token.renew-interval.sec 86400 - Renewal interval of a delagation token, in seconds. Default value 1 day. + Renewal interval of a delegation token, in seconds. Default value 1 day. @@ -640,7 +652,7 @@ $H4 HTTP Authentication Signature KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully. This HTTP Cookie has an expiration time, after which it will trigger a new authentication sequence. This is done to avoid triggering the authentication on every HTTP request of a client. -A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this all KMS instances must share the signing secret. +A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this, all KMS instances must share the signing secret. Please see [SignerSecretProvider Configuration](../hadoop-auth/Configuration.html#SignerSecretProvider_Configuration) for detailed description and configuration examples. Note that KMS configurations need to be prefixed with `hadoop.kms.authentication`, as shown in the example below. This secret sharing can be done using a Zookeeper service which is configured in KMS with the following properties in the `kms-site.xml`: @@ -650,8 +662,9 @@ This secret sharing can be done using a Zookeeper service which is configured in zookeeper Indicates how the secret to sign the authentication cookies will be - stored. Options are 'random' (default), 'string' and 'zookeeper'. + stored. Options are 'random' (default), 'file' and 'zookeeper'. If using a setup with multiple KMS instances, 'zookeeper' should be used. + If using file, signature.secret.file should be configured and point to the secret file. @@ -659,7 +672,7 @@ This secret sharing can be done using a Zookeeper service which is configured in /hadoop-kms/hadoop-auth-signature-secret The Zookeeper ZNode path where the KMS instances will store and retrieve - the secret from. + the secret from. All KMS instances that need to coordinate should point to the same path. @@ -696,7 +709,11 @@ This secret sharing can be done using a Zookeeper service which is configured in $H4 Delegation Tokens -TBD +Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens too. + +Under HA, A KMS instance must verify the delegation token given by another KMS instance, by checking the shared secret used to sign the delegation token. To do this, all KMS instances must be able to retrieve the shared secret from ZooKeeper. + +Please see the examples given in HTTP Authentication section to configure ZooKeeper for secret sharing. $H3 KMS HTTP REST API