Configuring Keystone¶

Sample Configuration Files¶

The etc/ folder distributed with keystone contains example configuration files for each Server application.

• etc/keystone.conf.sample
• etc/keystone-paste.ini
• etc/logging.conf.sample
• etc/default_catalog.templates
• etc/sso_callback_template.html

keystone.conf sections¶

The primary configuration file is organized into the following sections:

• [DEFAULT] - General configuration
• [assignment] - Assignment system driver configuration
• [auth] - Authentication plugin configuration
• [cache] - Caching layer configuration
• [catalog] - Service catalog driver configuration
• [credential] - Credential system driver configuration
• [domain_config] - Domain configuration
• [endpoint_filter] - Endpoint filtering configuration
• [endpoint_policy] - Endpoint policy configuration
• [federation] - Federation driver configuration
• [fernet_tokens] - Fernet token configuration
• [identity] - Identity system driver configuration
• [identity_mapping] - Identity mapping system driver configuration
• [ldap] - LDAP configuration options
• [memcache] - Memcache configuration options
• [oauth1] - OAuth 1.0a system driver configuration
• [paste_deploy] - Pointer to the PasteDeploy configuration file
• [policy] - Policy system driver configuration for RBAC
• [resource] - Resource system driver configuration
• [revoke] - Revocation system driver configuration
• [role] - Role system driver configuration
• [saml] - SAML configuration options
• [security_compliance] - Security compliance configuration
• [shadow_users] - Shadow user configuration
• [signing] - Cryptographic signatures for PKI based tokens
• [token] - Token driver & token provider configuration
• [tokenless_auth] - Tokenless authentication configuration
• [trust] - Trust configuration

The keystone primary configuration file is expected to be named keystone.conf. When starting keystone, you can specify a different configuration file to use with --config-file. If you do not specify a configuration file, keystone will look in the following directories for a configuration file, in order:

• ~/.keystone/
• ~/
• /etc/keystone/
• /etc/

PasteDeploy configuration file is specified by the config_file parameter in [paste_deploy] section of the primary configuration file. If the parameter is not an absolute path, then keystone looks for it in the same directories as above. If not specified, WSGI pipeline definitions are loaded from the primary configuration file.

Setting up projects, users, and roles¶

The keystone-manage bootstrap command will create a user, project and role, and will assign the newly created role to the newly created user on the newly created project. By default, the names of these new resources will be called admin.

The defaults may be overridden by calling --bootstrap-username, --bootstrap-project-name and --bootstrap-role-name. Each of these have an environment variable equivalent: OS_BOOTSTRAP_USERNAME, OS_BOOTSTRAP_PROJECT_NAME and OS_BOOTSTRAP_ROLE_NAME.

A user password must also be supplied. This can be passed in as either --bootstrap-password, or set as an environment variable using OS_BOOTSTRAP_PASSWORD.

Optionally, if specified by --bootstrap-public-url, --bootstrap-admin-url and/or --bootstrap-internal-url or the equivalent environment variables, the command will create an identity service with the specified endpoint information. You may also configure the --bootstrap-region-id and --bootstrap-service-name for the endpoints to your deployment’s requirements.

Minimally, keystone can be bootstrapped with:

$ keystone-manage bootstrap --bootstrap-password s3cr3t

Verbosely, keystone can be bootstrapped with:

$ keystone-manage bootstrap \
    --bootstrap-password s3cr3t \
    --bootstrap-username admin \
    --bootstrap-project-name admin \
    --bootstrap-role-name admin \
    --bootstrap-service-name keystone \
    --bootstrap-region-id RegionOne \
    --bootstrap-admin-url http://localhost:35357 \
    --bootstrap-public-url http://localhost:5000 \
    --bootstrap-internal-url http://localhost:5000

This will create an admin user with the admin role on the admin project. The user will have the password specified in the command. Note that both the user and the project will be created in the default domain. By not creating an endpoint in the catalog users will need to provide endpoint overrides to perform additional identity operations.

By creating an admin user and an identity endpoint deployers may authenticate to keystone and perform identity operations like creating additional services and endpoints using that admin user. This will preclude the need to ever use or configure the admin_token (described below).

To test a proper configuration, a user can use OpenStackClient CLI:

$ openstack project list --os-username admin --os-project-name admin \
    --os-user-domain-id default --os-project-domain-id default \
    --os-identity-api-version 3 --os-auth-url http://localhost:5000 \
    --os-password s3cr3t

Admin Token¶

For a default installation of Keystone, before you can use the REST API, you need to define an authorization token. This is configured in keystone.conf file under the section [DEFAULT]. In the sample file provided with the Keystone project, the line defining this token is:

[DEFAULT]
admin_token = ADMIN

A “shared secret” that can be used to bootstrap Keystone. This token does not represent a user, and carries no explicit authorization. To disable in production (highly recommended), remove AdminTokenAuthMiddleware from your paste application pipelines (for example, in keystone-paste.ini)

Setting up projects, users, and roles¶

You need to minimally define a project, user, and role to link the project and user as the most basic set of details to get other services authenticating and authorizing with Keystone.

You will also want to create service users for nova, glance, swift, etc. to be able to use to authenticate users against Keystone. The auth_token middleware supports using either the shared secret described above as admin_token or users for each service.

Creating Service Users¶

To configure the OpenStack services with service users, we need to create a project for all the services, and then users for each of the services. We then assign those service users an admin role on the service project. This allows them to validate tokens - and to authenticate and authorize other user requests.

Create a project for the services, typically named service (however, the name can be whatever you choose):

$ openstack project create service

Create service users for nova, glance, swift, and neutron (or whatever subset is relevant to your deployment):

$ openstack user create nova --password Sekr3tPass --project service

Repeat this for each service you want to enable.

Create an administrative role for the service accounts, typically named admin (however the name can be whatever you choose). For adding the administrative role to the service accounts, you’ll need to know the name of the role you want to add. If you don’t have it handy, you can look it up quickly with:

$ openstack role list

Once you have it, grant the administrative role to the service users.

$ openstack role add admin --project service --user nova

Defining Services¶

Keystone also acts as a service catalog to let other OpenStack systems know where relevant API endpoints exist for OpenStack Services. The OpenStack Dashboard, in particular, uses this heavily - and this must be configured for the OpenStack Dashboard to properly function.

The endpoints for these services are defined in a template, an example of which is in the project as the file etc/default_catalog.templates.

Keystone supports two means of defining the services, one is the catalog template, as described above - in which case everything is detailed in that template.

The other is a SQL backend for the catalog service, in which case after Keystone is online, you need to add the services to the catalog:

$ openstack service create compute --name nova \
                                --description "Nova Compute Service"
$ openstack service create ec2 --name ec2 \
                               --description "EC2 Compatibility Layer"
$ openstack service create image --name glance \
                                  --description "Glance Image Service"
$ openstack service create identity --name keystone \
                                    --description "Keystone Identity Service"
$ openstack service create object-store --name swift \
                                 --description "Swift Service"

Domain-specific Drivers¶

Keystone supports the option (disabled by default) to specify identity driver configurations on a domain by domain basis, allowing, for example, a specific domain to have its own LDAP or SQL server. This is configured by specifying the following options:

[identity]
domain_specific_drivers_enabled = True
domain_config_dir = /etc/keystone/domains

Setting domain_specific_drivers_enabled to True will enable this feature, causing keystone to look in the domain_config_dir for config files of the form:

keystone.<domain_name>.conf

Options given in the domain specific configuration file will override those in the primary configuration file for the specified domain only. Domains without a specific configuration file will continue to use the options from the primary configuration file.

Keystone also supports the ability to store the domain-specific configuration options in the keystone SQL database, managed via the Identity API, as opposed to using domain-specific configuration files.

This capability (which is disabled by default) is enabled by specifying the following options in the main keystone configuration file:

[identity]
domain_specific_drivers_enabled = true
domain_configurations_from_database = true

Once enabled, any existing domain-specific configuration files in the configuration directory will be ignored and only those domain-specific configuration options specified via the Identity API will be used.

Unlike the file-based method of specifying domain-specific configurations, options specified via the Identity API will become active without needing to restart the keystone server. For performance reasons, the current state of configuration options for a domain are cached in the keystone server, and in multi-process and multi-threaded keystone configurations, the new configuration options may not become active until the cache has timed out. The cache settings for domain config options can be adjusted in the general keystone configuration file (option cache_time in the domain_config group).

For existing installations that already use file-based domain-specific configurations who wish to migrate to the SQL-based approach, the keystone-manage command can be used to upload all configuration files to the SQL database:

$ keystone-manage domain_config_upload --all

Once uploaded, these domain-configuration options will be visible via the Identity API as well as applied to the domain-specific drivers. It is also possible to upload individual domain-specific configuration files by specifying the domain name:

$ keystone-manage domain_config_upload --domain-name DOMAINA

Due to the need for user and group IDs to be unique across an OpenStack installation and for keystone to be able to deduce which domain and backend to use from just a user or group ID, it dynamically builds a persistent identity mapping table from a public ID to the actual domain, local ID (within that backend) and entity type. The public ID is automatically generated by keystone when it first encounters the entity. If the local ID of the entity is from a backend that does not guarantee to generate UUIDs, a hash algorithm will generate a public ID for that entity, which is what will be exposed by keystone.

The use of a hash will ensure that if the public ID needs to be regenerated then the same public ID will be created. This is useful if you are running multiple keystones and want to ensure the same ID would be generated whichever server you hit.

While keystone will dynamically maintain the identity mapping, including removing entries when entities are deleted via the keystone, for those entities in backends that are managed outside of keystone (e.g. a read-only LDAP), keystone will not know if entities have been deleted and hence will continue to carry stale identity mappings in its table. While benign, keystone provides an ability for operators to purge the mapping table of such stale entries using the keystone-manage command, for example:

$ keystone-manage mapping_purge --domain-name DOMAINA --local-id abc@de.com

A typical usage would be for an operator to obtain a list of those entries in an external backend that had been deleted out-of-band to keystone, and then call keystone-manage to purge those entries by specifying the domain and local-id. The type of the entity (i.e. user or group) may also be specified if this is needed to uniquely identify the mapping.

Since public IDs can be regenerated with the correct generator implementation, if the details of those entries that have been deleted are not available, then it is safe to simply bulk purge identity mappings periodically, for example:

$ keystone-manage mapping_purge --domain-name DOMAINA

will purge all the mappings for DOMAINA. The entire mapping table can be purged with the following command:

$ keystone-manage mapping_purge --all

Generating public IDs in the first run may take a while, and most probably first API requests to fetch user list will fail by timeout. To prevent this, mapping_populate command should be executed. It should be executed right after LDAP has been configured or after mapping_purge.

$ keystone-manage mapping_populate --domain DOMAINA

Public ID Generators¶

Keystone supports a customizable public ID generator and it is specified in the [identity_mapping] section of the configuration file. Keystone provides a sha256 generator as default, which produces regenerable public IDs. The generator algorithm for public IDs is a balance between key size (i.e. the length of the public ID), the probability of collision and, in some circumstances, the security of the public ID. The maximum length of public ID supported by keystone is 64 characters, and the default generator (sha256) uses this full capability. Since the public ID is what is exposed externally by keystone and potentially stored in external systems, some installations may wish to make use of other generator algorithms that have a different trade-off of attributes. A different generator can be installed by configuring the following property:

• generator - identity mapping generator. Defaults to sha256 (implemented by keystone.identity.id_generators.sha256.Generator)

How to Implement an Authentication Plugin¶

All authentication plugins must extend the keystone.auth.plugins.base.AuthMethodHandler class and implement the authenticate() method. The authenticate() method expects the following parameters.

• context - keystone’s request context
• auth_payload - the content of the authentication for a given method
• auth_context - user authentication context, a dictionary shared by all plugins. It contains method_names and extras by default. method_names is a list and extras is a dictionary.

If successful, the authenticate() method must provide a valid user_id in auth_context and return None. method_name is used to convey any additional authentication methods in case authentication is for re-scoping. For example, if the authentication is for re-scoping, a plugin must append the previous method names into method_names. Also, a plugin may add any additional information into extras. Anything in extras will be conveyed in the token’s extras field.

If authentication requires multiple steps, the authenticate() method must return the payload in the form of a dictionary for the next authentication step.

If authentication is unsuccessful, the authenticate() method must raise a keystone.exception.Unauthorized exception.

Simply add the new plugin name to the methods list along with your plugin class configuration in the [auth] sections of the configuration file to deploy it.

If the plugin requires additional configurations, it may register its own section in the configuration file.

Plugins are invoked in the order in which they are specified in the methods attribute of the authentication request body. If multiple plugins are invoked, all plugins must succeed in order to for the entire authentication to be successful. Furthermore, all the plugins invoked must agree on the user_id in the auth_context.

The REMOTE_USER environment variable is only set from a containing webserver. However, to ensure that a user must go through other authentication mechanisms, even if this variable is set, remove external from the list of plugins specified in methods. This effectively disables external authentication. For more details, refer to External Authentication.

Token Persistence Driver¶

Keystone supports customizable token persistence drivers. These can be specified in the [token] section of the configuration file. Keystone provides two non-test persistence backends. These can be set with the [token] driver configuration option.

The drivers keystone provides are:

• sql - The SQL-based (default) token persistence engine. Implemented by keystone.token.persistence.backends.sql.Token

Token Provider¶

Keystone supports customizable token providers and it is specified in the [token] section of the configuration file. Keystone provides two token provider options (fernet and uuid, with fernet being the default). Users may register their own token provider by configuring the [token] provider property.

• fernet - A Fernet based token provider. Implemented by keystone.token.providers.fernet.Provider
• uuid - A UUID based token provider. Implemented by keystone.token.providers.uuid.Provider

[cache] configuration section¶

• enabled - enables/disables caching across all of keystone

• debug_cache_backend - enables more in-depth logging from the cache backend (get, set, delete, etc)

• backend - the caching backend module to use e.g. dogpile.cache.memcached

  Note

  A given backend must be registered with dogpile.cache before it can be used. The default backend is the keystone no-op backend (keystone.common.cache.noop). If caching is desired a different backend will need to be specified. Current functional backends are:

  • dogpile.cache.memcached - Memcached backend using the standard python-memcached library (recommended for use with Apache httpd with mod_wsgi)

  • dogpile.cache.pylibmc - Memcached backend using the pylibmc library

  • dogpile.cache.bmemcached - Memcached using python-binary-memcached library.

  • dogpile.cache.redis - Redis backend

  • dogpile.cache.dbm - local DBM file backend

  • dogpile.cache.memory - in-memory cache

  • oslo_cache.mongo - MongoDB as caching backend

  • oslo_cache.memcache_pool - Memcache with pooling. This implementation also provides client connection re-use.

    Warning

    dogpile.cache.memory is not suitable for use outside of unit testing as it does not cleanup its internal cache on cache expiration, does not provide isolation to the cached data (values in the store can be inadvertently changed without extra layers of data protection added), and does not share cache between processes. This means that caching and cache invalidation will not be consistent or reliable when using keystone and the dogpile.cache.memory backend under any real workload.

• expiration_time - int, the default length of time to cache a specific value. A value of 0 indicates to not cache anything. It is recommended that the enabled option be used to disable cache instead of setting this to 0.

• backend_argument - an argument passed to the backend when instantiated backend_argument should be specified once per argument to be passed to the backend and in the format of <argument name>:<argument value>. e.g.: backend_argument = host:localhost

• proxies - comma delimited list of ProxyBackends e.g. my.example.Proxy, my.example.Proxy2

Current keystone systems that have caching capabilities:

• token

  The token system has a separate cache_time configuration option, that can be set to a value above or below the global expiration_time default, allowing for different caching behavior from the other systems in keystone. This option is set in the [token] section of the configuration file.

  The Token Revocation List cache time is handled by the configuration option revocation_cache_time in the [token] section. The revocation list is refreshed whenever a token is revoked. It typically sees significantly more requests than specific token retrievals or token validation calls.

• resource

  The resource system has a separate cache_time configuration option, that can be set to a value above or below the global expiration_time default, allowing for different caching behavior from the other systems in keystone. This option is set in the [resource] section of the configuration file.

  Currently resource has caching for project and domain specific requests (primarily around the CRUD actions). The list_projects and list_domains methods are not subject to caching.

  Warning

  Be aware that if a read-only resource backend is in use, the cache will not immediately reflect changes on the back end. Any given change may take up to the cache_time (if set in the [resource] section of the configuration) or the global expiration_time (set in the [cache] section of the configuration) before it is reflected. If this type of delay (when using a read-only resource backend) is an issue, it is recommended that caching be disabled on resource. To disable caching specifically on resource, in the [resource] section of the configuration set caching to False.

• role

  Currently role has caching for get_role, but not for list_roles. The role system has a separate cache_time configuration option, that can be set to a value above or below the global expiration_time default, allowing for different caching behavior from the other systems in keystone. This option is set in the [role] section of the configuration file.

  Warning

  Be aware that if a read-only role backend is in use, the cache will not immediately reflect changes on the back end. Any given change may take up to the cache_time (if set in the [role] section of the configuration) or the global expiration_time (set in the [cache] section of the configuration) before it is reflected. If this type of delay (when using a read-only role backend) is an issue, it is recommended that caching be disabled on role. To disable caching specifically on role, in the [role] section of the configuration set caching to False.

For more information about the different backends (and configuration options):

• dogpile.cache.backends.memory
• dogpile.cache.backends.memcached
• dogpile.cache.backends.redis
• dogpile.cache.backends.file

Signing Certificate Issued by External CA¶

You may use a signing certificate issued by an external CA instead of generated by keystone-manage. However, certificate issued by external CA must satisfy the following conditions:

• all certificate and key files must be in Privacy Enhanced Mail (PEM) format
• private key files must not be protected by a password

The basic workflow for using a signing certificate issued by an external CA involves:

1. Request Signing Certificate from External CA
2. Convert certificate and private key to PEM if needed
3. Install External Signing Certificate

Request Signing Certificate from External CA¶

One way to request a signing certificate from an external CA is to first generate a PKCS #10 Certificate Request Syntax (CRS) using OpenSSL CLI.

First create a certificate request configuration file (e.g. cert_req.conf):

[ req ]
default_bits            = 2048
default_keyfile         = keystonekey.pem
default_md              = default

prompt                  = no
distinguished_name      = distinguished_name

[ distinguished_name ]
countryName             = US
stateOrProvinceName     = CA
localityName            = Sunnyvale
organizationName        = OpenStack
organizationalUnitName  = Keystone
commonName              = Keystone Signing
emailAddress            = keystone@openstack.org

Then generate a CRS with OpenSSL CLI. Do not encrypt the generated private key. The -nodes option must be used.

For example:

$ openssl req -newkey rsa:2048 -keyout signing_key.pem -keyform PEM -out signing_cert_req.pem -outform PEM -config cert_req.conf -nodes

If everything is successfully, you should end up with signing_cert_req.pem and signing_key.pem. Send signing_cert_req.pem to your CA to request a token signing certificate and make sure to ask the certificate to be in PEM format. Also, make sure your trusted CA certificate chain is also in PEM format.

Install External Signing Certificate¶

Assuming you have the following already:

• signing_cert.pem - (Keystone token) signing certificate in PEM format
• signing_key.pem - corresponding (non-encrypted) private key in PEM format
• cacert.pem - trust CA certificate chain in PEM format

Copy the above to your certificate directory. For example:

$ mkdir -p /etc/keystone/ssl/certs
$ cp signing_cert.pem /etc/keystone/ssl/certs/
$ cp signing_key.pem /etc/keystone/ssl/certs/
$ cp cacert.pem /etc/keystone/ssl/certs/
$ chmod -R 700 /etc/keystone/ssl/certs

Make sure the certificate directory is root-protected.

If your certificate directory path is different from the default /etc/keystone/ssl/certs, make sure it is reflected in the [signing] section of the configuration file.

SQL-based Service Catalog (sql.Catalog)¶

A dynamic database-backed driver fully supporting persistent configuration.

keystone.conf example:

[catalog]
driver = sql

To build your service catalog using this driver, see the built-in help:

$ openstack --help
$ openstack service create --help
$ openstack endpoint create --help

File-based Service Catalog (templated.Catalog)¶

The templated catalog is an in-memory backend initialized from a read-only template_file. Choose this option only if you know that your service catalog will not change very much over time.

keystone.conf example:

[catalog]
driver = templated
template_file = /opt/stack/keystone/etc/default_catalog.templates

The value of template_file is expected to be an absolute path to your service catalog configuration. An example template_file is included in keystone, however you should create your own to reflect your deployment.

Step 1: Configure keystone.conf¶

Ensure that your keystone.conf is configured to use a SQL driver:

[identity]
driver = sql

You may also want to configure your [database] settings to better reflect your environment:

[database]
connection = sqlite:///keystone.db
idle_timeout = 200

Step 2: Sync your new, empty database¶

You should now be ready to initialize your new database without error, using:

$ keystone-manage db_sync

To test this, you should now be able to start keystone:

$ uwsgi --http 127.0.0.1:35357 --wsgi-file $(which keystone-wsgi-admin)

And use the OpenStack Client to list your projects (which should successfully return an empty list from your new database):

$ openstack --os-token ADMIN --os-url http://127.0.0.1:35357/v2.0/ project list

General keystone-manage options:¶

• --help : display verbose help output.

Invoking keystone-manage by itself will give you some usage information.

Available commands:

• bootstrap: Perform the basic bootstrap process.
• credential_migrate: Encrypt credentials using a new primary key.
• credential_rotate: Rotate Fernet keys for credential encryption.
• credential_setup: Setup a Fernet key repository for credential encryption.
• db_sync: Sync the database.
• db_version: Print the current migration version of the database.
• doctor: Diagnose common problems with keystone deployments.
• domain_config_upload: Upload domain configuration file.
• fernet_rotate: Rotate keys in the Fernet key repository.
• fernet_setup: Setup a Fernet key repository for token encryption.
• mapping_populate: Prepare domain-specific LDAP backend.
• mapping_purge: Purge the identity mapping table.
• mapping_engine: Test your federation mapping rules.
• saml_idp_metadata: Generate identity provider metadata.
• token_flush: Purge expired tokens.

Authenticating with a Password via CLI¶

To authenticate with keystone using a password and python-openstackclient, set the following flags, note that the following user referenced below should be granted the admin role.

• --os-username OS_USERNAME: Name of your user
• --os-password OS_PASSWORD: Password for your user
• --os-project-name OS_PROJECT_NAME: Name of your project
• --os-auth-url OS_AUTH_URL: URL of the keystone authentication server

You can also set these variables in your environment so that they do not need to be passed as arguments each time:

$ export OS_USERNAME=my_username
$ export OS_PASSWORD=my_password
$ export OS_PROJECT_NAME=my_project
$ export OS_AUTH_URL=http://localhost:35357/v2.0

For example, the commands user list, token issue and project create can be invoked as follows:

# Using password authentication, with environment variables
$ export OS_USERNAME=admin
$ export OS_PASSWORD=secret
$ export OS_PROJECT_NAME=admin
$ export OS_AUTH_URL=http://localhost:35357/v2.0
$ openstack user list
$ openstack project create demo
$ openstack token issue

# Using password authentication, with flags
$ openstack --os-username=admin --os-password=secret --os-project-name=admin --os-auth-url=http://localhost:35357/v2.0 user list
$ openstack --os-username=admin --os-password=secret --os-project-name=admin --os-auth-url=http://localhost:35357/v2.0 project create demo

Debugging LDAP¶

For additional information on LDAP connections, performance (such as slow response time), or field mappings, setting debug_level in the [ldap] section is used to enable debugging:

debug_level = 4095

This setting in turn sets OPT_DEBUG_LEVEL in the underlying python library. This field is a bit mask (integer), and the possible flags are documented in the OpenLDAP manpages. Commonly used values include 255 and 4095, with 4095 being more verbose.

Enabled Emulation¶

Some directory servers do not provide any enabled attribute. For these servers, the user_enabled_emulation attribute has been created. It is enabled by setting the respective flags to True. Then the attribute user_enabled_emulation_dn may be set to specify how the enabled users are selected. This attribute works by using a groupOfNames entry and adding whichever users or that you want enabled to the respective group with the member attribute. For example, this will mark any user who is a member of enabled_users as enabled:

[ldap]
user_enabled_emulation = True
user_enabled_emulation_dn = cn=enabled_users,cn=groups,dc=openstack,dc=org

The default values for user enabled emulation DN is cn=enabled_users,$user_tree_dn.

If a different LDAP schema is used for group membership, it is possible to use the group_objectclass and group_member_attribute attributes to determine membership in the enabled emulation group by setting the user_enabled_emulation_use_group_config attribute to True.

Secure Connection¶

If you are using a directory server to provide the Identity service, it is strongly recommended that you utilize a secure connection from keystone to the directory server. In addition to supporting LDAP, keystone also provides Transport Layer Security (TLS) support. There are some basic configuration options for enabling TLS, identifying a single file or directory that contains certificates for all the Certificate Authorities that the keystone LDAP client will recognize, and declaring what checks the client should perform on server certificates. This functionality can easily be configured as follows:

[ldap]
use_tls = True
tls_cacertfile = /etc/keystone/ssl/certs/cacert.pem
tls_cacertdir = /etc/keystone/ssl/certs/
tls_req_cert = demand

A few points worth mentioning regarding the above options. If both tls_cacertfile and tls_cacertdir are set then tls_cacertfile will be used and tls_cacertdir is ignored. Furthermore, valid options for tls_req_cert are demand, never, and allow. These correspond to the standard options permitted by the TLS_REQCERT TLS option.

Read Only LDAP¶

Many environments typically have user and group information in directories that are accessible by LDAP. This information is for read-only use in a wide array of applications. Prior to the Havana release, we could not deploy keystone with read-only directories as backends because keystone also needed to store information such as projects, roles, domains and role assignments into the directories in conjunction with reading user and group information.

Keystone now provides an option whereby these read-only directories can be easily integrated as it now enables its identity entities (which comprises users, groups, and group memberships) to be served out of directories while resource (which comprises projects and domains), assignment and role entities are to be served from different keystone backends (i.e. SQL). To enable this option, you must have the following keystone.conf options set:

[identity]
driver = ldap

[resource]
driver = sql

[assignment]
driver = sql

[role]
driver = sql

With the above configuration, keystone will only lookup identity related information such users, groups, and group membership from the directory, while resources, roles and assignment related information will be provided by the SQL backend. Also note that if there is an LDAP Identity, and no resource, assignment or role backend is specified, they will default to LDAP. Although this may seem counter intuitive, it is provided for backwards compatibility. Nonetheless, the explicit option will always override the implicit option, so specifying the options as shown above will always be correct.

Connection Pooling¶

Various LDAP backends in keystone use a common LDAP module to interact with LDAP data. By default, a new connection is established for each LDAP operation. This can become highly expensive when TLS support is enabled, which is a likely configuration in an enterprise setup. Reuse of connectors from a connection pool drastically reduces overhead of initiating a new connection for every LDAP operation.

Keystone provides connection pool support via configuration. This will keep LDAP connectors alive and reused for subsequent LDAP operations. The connection lifespan is configurable as other pooling specific attributes.

In the LDAP identity driver, keystone authenticates end users via an LDAP bind with the user’s DN and provided password. This kind of authentication bind can fill up the pool pretty quickly, so a separate pool is provided for end user authentication bind calls. If a deployment does not want to use a pool for those binds, then it can disable pooling selectively by setting use_auth_pool to false. If a deployment wants to use a pool for those authentication binds, then use_auth_pool needs to be set to true. For the authentication pool, a different pool size (auth_pool_size) and connection lifetime (auth_pool_connection_lifetime) can be specified. With an enabled authentication pool, its connection lifetime should be kept short so that the pool frequently re-binds the connection with the provided credentials and works reliably in the end user password change case. When use_pool is false (disabled), then the authentication pool configuration is also not used.

Connection pool configuration is part of the [ldap] configuration section:

[ldap]
# Enable LDAP connection pooling for queries to the LDAP server. There is
# typically no reason to disable this. (boolean value)
use_pool = true

# The size of the LDAP connection pool. This option has no effect unless
# `[ldap] use_pool` is also enabled. (integer value)
# Minimum value: 1
pool_size = 10

# The maximum number of times to attempt reconnecting to the LDAP server before
# aborting. A value of zero prevents retries. This option has no effect unless
# `[ldap] use_pool` is also enabled. (integer value)
# Minimum value: 0
pool_retry_max = 3

# The number of seconds to wait before attempting to reconnect to the LDAP
# server. This option has no effect unless `[ldap] use_pool` is also enabled.
# (floating point value)
pool_retry_delay = 0.1

# The connection timeout to use with the LDAP server. A value of `-1` means
# that connections will never timeout. This option has no effect unless `[ldap]
# use_pool` is also enabled. (integer value)
# Minimum value: -1
pool_connection_timeout = -1

# The maximum connection lifetime to the LDAP server in seconds. When this
# lifetime is exceeded, the connection will be unbound and removed from the
# connection pool. This option has no effect unless `[ldap] use_pool` is also
# enabled. (integer value)
# Minimum value: 1
pool_connection_lifetime = 600

# Enable LDAP connection pooling for end user authentication. There is
# typically no reason to disable this. (boolean value)
use_auth_pool = true

# The size of the connection pool to use for end user authentication. This
# option has no effect unless `[ldap] use_auth_pool` is also enabled. (integer
# value)
# Minimum value: 1
auth_pool_size = 100

# The maximum end user authentication connection lifetime to the LDAP server in
# seconds. When this lifetime is exceeded, the connection will be unbound and
# removed from the connection pool. This option has no effect unless `[ldap]
# use_auth_pool` is also enabled. (integer value)
# Minimum value: 1
auth_pool_connection_lifetime = 60

Specifying Multiple LDAP servers¶

Multiple LDAP server URLs can be provided to keystone to provide high-availability support for a single LDAP backend. To specify multiple LDAP servers, simply change the url option in the [ldap] section. The new option should list the different servers, each separated by a comma. For example:

[ldap]
url = "ldap://localhost,ldap://backup.localhost"

Configuring credential encryption¶

The configuration for credential encryption is straightforward. There are only two configuration options needed:

[credential]
provider = fernet
key_repository = /etc/keystone/credential-keys/

[credential] provider defaults to the only option supplied by keystone, fernet. There is no reason to change this option unless you wish to provide a custom credential encryption implementation. The [credential] key_repository location is a requirement of using fernet but will default to the /etc/keystone/credential-keys/ directory. Both [credential] key_repository and [fernet_tokens] key_repository define locations for keys used to encrypt things. One holds the keys to encrypt and decrypt credentials and the other holds keys to encrypt and decrypt tokens. It is imperative that these repositories are managed separately and they must not share keys. Meaning they cannot share the same directory path. The [credential] key_repository is only allowed to have three keys. This is not configurable and allows for credentials to be re-encrypted periodically with a new encryption key for the sake of security.

How credential encryption works¶

The implementation of this feature did not change any existing credential API contracts. All changes are transparent to the user unless you’re inspecting the credential backend directly.

When creating a credential, keystone will encrypt the blob attribute before persisting it to the backend. Keystone will also store a hash of the key that was used to encrypt the information in that credential. Since Fernet is used to encrypt credentials, a key repository consists of multiple keys. Keeping track of which key was used to encrypt each credential is an important part of encryption key management. Why this is important is detailed later in the Encryption key management section.

When updating an existing credential’s blob attribute, keystone will encrypt the new blob and update the key hash.

When listing or showing credentials, all blob attributes are decrypted in the response. Neither the cipher text, nor the hash of the key used to encrypt the blob are exposed through the API. Furthermore, the key is only used internally to keystone.

Encrypting existing credentials¶

When upgrading a Mitaka deployment to Newton, three database migrations will ensure all credentials are encrypted. The process is as follows:

1. An additive schema change is made to create the new encrypted_blob and key_hash columns in the existing credential table using keystone-manage db_sync --expand.
2. A data migration will loop through all existing credentials, encrypt each blob and store the result in the new encrypted_blob column. The hash of the key used is also written to the key_hash column for that specific credential. This step is done using keystone-manage db_sync --migrate.
3. A contractive schema will remove the blob column that held the plain text representations of the credential using keystone-manage db_sync --contract. This should only be done after all nodes in the deployment are running Newton. If any Mitaka nodes are running after the database is contracted, they won’t be able to read credentials since they are looking for the blob column that no longer exists.

If performing a rolling upgrade, please note that a limited service outage will take affect during this migration. When the migration is in place, credentials will become read-only until the database is contracted. After the contract phase is complete, credentials will be writeable to the backend. A [credential] key_repository location must be specified through configuration and bootstrapped with keys using keystone-manage credential_setup prior to migrating any existing credentials. If a new key repository isn’t setup using keystone-manage credential_setup keystone will assume a null key to encrypt and decrypt credentials until a proper key repository is present. The null key is a key consisting of all null bytes and its only purpose is to ease the upgrade process from Mitaka to Newton. It is highly recommended that the null key isn’t used. It is no more secure than storing credentials in plain text. If the null key is used, you should migrate to a proper key repository using keystone-manage credential_setup and keystone-manage credential_migrate.

Encryption key management¶

Key management of [credential] key_repository is handled with three keystone-manage commands:

1. keystone-manage credential_setup
2. keystone-manage credential_rotate
3. keystone-manage credential_migrate

keystone-manage credential_setup will populate [credential] key_repository with new encryption keys. This must be done in order for proper credential encryption to work, with the exception of the null key. This step should only be done once.

keystone-manage credential_rotate will create and rotate a new encryption key in the [credential] key_repository. This will only be done if all credential key hashes match the hash of the current primary key. If any credential has been encrypted with an older key, or secondary key, the rotation will fail. Failing the rotation is necessary to prevent overrotation, which would leave some credentials indecipherable since the key used to encrypt it no longer exists. If this step fails, it is possible to forcibly re-key all credentials using the same primary key with keystone-manage credential_migrate.

keystone-manage credential_migrate will check the backend for credentials whose key hash doesn’t match the hash of the current primary key. Any credentials with a key hash mismatching the current primary key will be re-encrypted with the current primary key. The new cipher text and key hash will be updated in the backend.