Middleware¶

Account Quotas¶

account_quotas is a middleware which blocks write requests (PUT, POST) if a given account quota (in bytes) is exceeded while DELETE requests are still allowed.

account_quotas uses the x-account-meta-quota-bytes metadata entry to store the quota. Write requests to this metadata entry are only permitted for resellers. There is no quota limit if x-account-meta-quota-bytes is not set.

The account_quotas middleware should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. For example:

[pipeline:main]
pipeline = catch_errors cache tempauth account_quotas proxy-server

[filter:account_quotas]
use = egg:swift#account_quotas

To set the quota on an account:

swift -A http://127.0.0.1:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:10000

Remove the quota:

swift -A http://127.0.0.1:8080/auth/v1.0 -U account:reseller -K secret post -m quota-bytes:

The same limitations apply for the account quotas as for the container quotas.

For example, when uploading an object without a content-length header the proxy server doesn’t know the final size of the currently uploaded object and the upload will be allowed if the current account size is within the quota. Due to the eventual consistency further uploads might be possible until the account size has been updated.

class swift.common.middleware.account_quotas.AccountQuotaMiddleware(app, *args, **kwargs)¶

Bases: object

Account quota middleware

See above for a full description.

swift.common.middleware.account_quotas.filter_factory(global_conf, **local_conf)¶

Returns a WSGI filter app for use with paste.deploy.

AWS S3 Api¶

The s3api middleware will emulate the S3 REST api on top of swift.

To enable this middleware to your configuration, add the s3api middleware in front of the auth middleware. See proxy-server.conf-sample for more detail and configurable options.

To set up your client, ensure you are using the tempauth or keystone auth system for swift project. When your swift on a SAIO environment, make sure you have setting the tempauth middleware configuration in proxy-server.conf, and the access key will be the concatenation of the account and user strings that should look like test:tester, and the secret access key is the account password. The host should also point to the swift storage hostname.

The tempauth option example:

[filter:tempauth]
use = egg:swift#tempauth
user_admin_admin = admin .admin .reseller_admin
user_test_tester = testing

An example client using tempauth with the python boto library is as follows:

from boto.s3.connection import S3Connection
connection = S3Connection(
    aws_access_key_id='test:tester',
    aws_secret_access_key='testing',
    port=8080,
    host='127.0.0.1',
    is_secure=False,
    calling_format=boto.s3.connection.OrdinaryCallingFormat())

And if you using keystone auth, you need the ec2 credentials, which can be downloaded from the API Endpoints tab of the dashboard or by openstack ec2 command.

Here is showing to create an EC2 credential:

# openstack ec2 credentials create
+------------+---------------------------------------------------+
| Field      | Value                                             |
+------------+---------------------------------------------------+
| access     | c2e30f2cd5204b69a39b3f1130ca8f61                  |
| links      | {u'self': u'http://controller:5000/v3/......'}    |
| project_id | 407731a6c2d0425c86d1e7f12a900488                  |
| secret     | baab242d192a4cd6b68696863e07ed59                  |
| trust_id   | None                                              |
| user_id    | 00f0ee06afe74f81b410f3fe03d34fbc                  |
+------------+---------------------------------------------------+

An example client using keystone auth with the python boto library will be:

from boto.s3.connection import S3Connection
connection = S3Connection(
    aws_access_key_id='c2e30f2cd5204b69a39b3f1130ca8f61',
    aws_secret_access_key='baab242d192a4cd6b68696863e07ed59',
    port=8080,
    host='127.0.0.1',
    is_secure=False,
    calling_format=boto.s3.connection.OrdinaryCallingFormat())

[pipeline:main]
pipeline = proxy-logging cache s3api tempauth bulk slo proxy-logging proxy-server

[pipeline:main]
pipeline = proxy-logging cache authtoken s3api s3token keystoneauth bulk slo proxy-logging proxy-server

[filter:s3api]
use = egg:swift#s3api

class swift.common.middleware.s3api.s3request.HashingInput(reader, content_length, hasher, expected_hex_hash)¶

Bases: object

wsgi.input wrapper to verify the hash of the input as it’s read.

class swift.common.middleware.s3api.s3request.S3AclRequest(env, app, slo_enabled=True, storage_domain='', location='US', force_request_log=False, dns_compliant_bucket_names=True, allow_multipart_uploads=True, allow_no_owner=False)¶

Bases: swift.common.middleware.s3api.s3request.S3Request

S3Acl request object.

authenticate(app)¶

authenticate method will run pre-authenticate request and retrieve account information. Note that it currently supports only keystone and tempauth. (no support for the third party authentication middleware)

get_acl_response(app, method=None, container=None, obj=None, headers=None, body=None, query=None)¶

Wrapper method of _get_response to add s3 acl information from response sysmeta headers.

get_response(app, method=None, container=None, obj=None, headers=None, body=None, query=None)¶

Wrap up get_response call to hook with acl handling method.

to_swift_req(method, container, obj, query=None, body=None, headers=None)¶

Create a Swift request based on this request’s environment.

class swift.common.middleware.s3api.s3request.S3Request(env, app=None, slo_enabled=True, storage_domain='', location='US', force_request_log=False, dns_compliant_bucket_names=True, allow_multipart_uploads=True, allow_no_owner=False)¶

Bases: swift.common.swob.Request

S3 request object.

body¶

swob.Request.body is not secure against malicious input. It consumes too much memory without any check when the request body is excessively large. Use xml() instead.

bucket_acl¶

Get and set the container acl property

check_copy_source(app)¶

check_copy_source checks the copy source existence and if copying an object to itself, for illegal request parameters

Returns:	the source HEAD response

get_container_info(app)¶

get_container_info will return a result dict of get_container_info from the backend Swift.

Returns:	a dictionary of container info from swift.controllers.base.get_container_info
Raises:	NoSuchBucket when the container doesn’t exist
Raises:	InternalError when the request failed without 404

get_response(app, method=None, container=None, obj=None, headers=None, body=None, query=None)¶

get_response is an entry point to be extended for child classes. If additional tasks needed at that time of getting swift response, we can override this method. swift.common.middleware.s3api.s3request.S3Request need to just call _get_response to get pure swift response.

object_acl¶

Get and set the object acl property

timestamp¶

S3Timestamp from Date header. If X-Amz-Date header specified, it will be prior to Date header.

:return : S3Timestamp instance

to_swift_req(method, container, obj, query=None, body=None, headers=None)¶

Create a Swift request based on this request’s environment.

xml(max_length)¶

Similar to swob.Request.body, but it checks the content length before creating a body string.

class swift.common.middleware.s3api.s3request.SigV4Mixin¶

Bases: object

A request class mixin to provide S3 signature v4 functionality

timestamp¶

Return timestamp string according to the auth type The difference from v2 is v4 have to see ‘X-Amz-Date’ even though it’s query auth type.

class swift.common.middleware.s3api.s3request.SigV4Request(env, app=None, slo_enabled=True, storage_domain='', location='US', force_request_log=False, dns_compliant_bucket_names=True, allow_multipart_uploads=True, allow_no_owner=False)¶

Bases: swift.common.middleware.s3api.s3request.SigV4Mixin, swift.common.middleware.s3api.s3request.S3Request

class swift.common.middleware.s3api.s3request.SigV4S3AclRequest(env, app, slo_enabled=True, storage_domain='', location='US', force_request_log=False, dns_compliant_bucket_names=True, allow_multipart_uploads=True, allow_no_owner=False)¶

Bases: swift.common.middleware.s3api.s3request.SigV4Mixin, swift.common.middleware.s3api.s3request.S3AclRequest

swift.common.middleware.s3api.s3request.get_request_class(env, s3_acl)¶

Helper function to find a request class to use from Map

exception swift.common.middleware.s3api.s3response.AccessDenied(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.AccountProblem(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.AmbiguousGrantByEmailAddress(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.AuthorizationHeaderMalformed(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.AuthorizationQueryParametersError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.BadDigest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.BucketAlreadyExists(bucket, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.BucketAlreadyOwnedByYou(bucket, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.BucketNotEmpty(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.CredentialsNotSupported(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.CrossLocationLoggingProhibited(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.EntityTooLarge(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.EntityTooSmall(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.ErrorResponse(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.S3ResponseBase, swift.common.swob.HTTPException

S3 error object.

Reference information about S3 errors is available at: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

exception swift.common.middleware.s3api.s3response.ExpiredToken(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

class swift.common.middleware.s3api.s3response.HeaderKey¶

Bases: str

A string object that normalizes string as S3 clients expect with title().

title() → str¶

Return a titlecased version of S, i.e. words start with title case characters, all remaining cased characters have lower case.

class swift.common.middleware.s3api.s3response.HeaderKeyDict(base_headers=None, **kwargs)¶

Bases: swift.common.header_key_dict.HeaderKeyDict

Similar to the HeaderKeyDict class in Swift, but its key name is normalized as S3 clients expect.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.¶

pop(k[, d]) → v, remove specified key and return the corresponding value.¶

If key is not found, d is returned if given, otherwise KeyError is raised

exception swift.common.middleware.s3api.s3response.IllegalVersioningConfigurationException(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.IncompleteBody(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.IncorrectNumberOfFilesInPostRequest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InlineDataTooLarge(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InternalError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidAccessKeyId(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidArgument(name, value, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidBucketName(bucket, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidBucketState(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidDigest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidLocationConstraint(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidObjectState(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidPart(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidPartOrder(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidPayer(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidPolicyDocument(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidRange(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidRequest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidSOAPRequest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidSecurity(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidStorageClass(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidTargetBucketForLogging(bucket, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidToken(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.InvalidURI(uri, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.KeyTooLong(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MalformedACLError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MalformedPOSTRequest(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MalformedXML(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MaxMessageLengthExceeded(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MaxPostPreDataLengthExceededError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MetadataTooLarge(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MethodNotAllowed(method, resource_type, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MissingContentLength(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MissingRequestBodyError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MissingSecurityElement(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.MissingSecurityHeader(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoLoggingStatusForKey(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoSuchBucket(bucket, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoSuchKey(key, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoSuchLifecycleConfiguration(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoSuchUpload(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NoSuchVersion(key, version_id, msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NotSignedUp(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.NotSuchBucketPolicy(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.OperationAborted(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.PermanentRedirect(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.PreconditionFailed(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.Redirect(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.RequestIsNotMultiPartContent(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.RequestTimeTooSkewed(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.RequestTimeout(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.RequestTorrentOfBucketError(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.RestoreAlreadyInProgress(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.S3NotImplemented(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

class swift.common.middleware.s3api.s3response.S3Response(*args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.S3ResponseBase, swift.common.swob.Response

Similar to the Response class in Swift, but uses our HeaderKeyDict for headers instead of Swift’s HeaderKeyDict. This also translates Swift specific headers to S3 headers.

classmethod from_swift_resp(sw_resp)¶

Create a new S3 response object based on the given Swift response.

class swift.common.middleware.s3api.s3response.S3ResponseBase¶

Bases: object

Base class for swift3 responses.

exception swift.common.middleware.s3api.s3response.ServiceUnavailable(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.SignatureDoesNotMatch(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.SlowDown(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.TemporaryRedirect(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.TokenRefreshRequired(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.TooManyBuckets(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.UnexpectedContent(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.UnresolvableGrantByEmailAddress(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.s3response.UserKeyMustBeSpecified(msg=None, *args, **kwargs)¶

Bases: swift.common.middleware.s3api.s3response.ErrorResponse

exception swift.common.middleware.s3api.exception.ACLError¶

Bases: swift.common.middleware.s3api.exception.S3Exception

exception swift.common.middleware.s3api.exception.BadSwiftRequest¶

Bases: swift.common.middleware.s3api.exception.S3Exception

exception swift.common.middleware.s3api.exception.InvalidSubresource(resource, cause)¶

Bases: swift.common.middleware.s3api.exception.S3Exception

exception swift.common.middleware.s3api.exception.NotS3Request¶

Bases: swift.common.middleware.s3api.exception.S3Exception

exception swift.common.middleware.s3api.exception.S3Exception¶

Bases: Exception

exception swift.common.middleware.s3api.etree.DocumentInvalid¶

Bases: swift.common.middleware.s3api.exception.S3Exception

exception swift.common.middleware.s3api.etree.XMLSyntaxError¶

Bases: swift.common.middleware.s3api.exception.S3Exception

class swift.common.middleware.s3api.utils.Config(base=None)¶

Bases: dict

update([E, ]**F) → None. Update D from dict/iterable E and F.¶

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

class swift.common.middleware.s3api.utils.S3Timestamp(timestamp, offset=0, delta=0)¶

Bases: swift.common.utils.Timestamp

amz_date_format¶

this format should be like ‘YYYYMMDDThhmmssZ’

swift.common.middleware.s3api.utils.mktime(timestamp_str, time_format='%Y-%m-%dT%H:%M:%S')¶

mktime creates a float instance in epoch time really like as time.mktime

the difference from time.mktime is allowing to 2 formats string for the argument for the S3 testing usage. TODO: support

Parameters:	timestamp_str – a string of timestamp formatted as (a) RFC2822 (e.g. date header) (b) %Y-%m-%dT%H:%M:%S (e.g. copy result) time_format – a string of format to parse in (b) process
Returns:	a float instance in epoch time

swift.common.middleware.s3api.utils.sysmeta_header(resource, name)¶

Returns the system metadata header for given resource type and name.

swift.common.middleware.s3api.utils.sysmeta_prefix(resource)¶

Returns the system metadata prefix for given resource type.

swift.common.middleware.s3api.utils.validate_bucket_name(name, dns_compliant_bucket_names)¶

Validates the name of the bucket against S3 criteria, http://docs.amazonwebservices.com/AmazonS3/latest/BucketRestrictions.html True is valid, False is invalid.

AccessControlPolicy:
    Owner:
    AccessControlList:
        Grant[n]:
            (Grantee, Permission)

class BucketAclHandler(BaseAclHandler):
    def PUT:
        << put acl handling algorithms here for PUT bucket >>

swift.common.middleware.s3api.acl_utils.handle_acl_header(req)¶

Handle the x-amz-acl header. Note that this header currently used for only normal-acl (not implemented) on s3acl. TODO: add translation to swift acl like as x-container-read to s3acl

swift.common.middleware.s3api.acl_utils.swift_acl_translate(acl, group='', user='', xml=False)¶

Takes an S3 style ACL and returns a list of header/value pairs that implement that ACL in Swift, or “NotImplemented” if there isn’t a way to do that yet.

class swift.common.middleware.s3api.controllers.base.Controller(app, conf, logger, **kwargs)¶

Bases: object

Base WSGI controller class for the middleware

classmethod resource_type()¶

Returns the target resource type of this controller.

class swift.common.middleware.s3api.controllers.base.UnsupportedController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles unsupported requests.

swift.common.middleware.s3api.controllers.base.bucket_operation(func=None, err_resp=None, err_msg=None)¶

A decorator to ensure that the request is a bucket operation. If the target resource is an object, this decorator updates the request by default so that the controller handles it as a bucket operation. If ‘err_resp’ is specified, this raises it on error instead.

swift.common.middleware.s3api.controllers.base.check_container_existence(func)¶

A decorator to ensure the container existence.

swift.common.middleware.s3api.controllers.base.object_operation(func)¶

A decorator to ensure that the request is an object operation. If the target resource is not an object, this raises an error response.

class swift.common.middleware.s3api.controllers.service.ServiceController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles account level requests.

GET(req)¶

Handle GET Service request

class swift.common.middleware.s3api.controllers.bucket.BucketController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles bucket request.

DELETE(req)¶

Handle DELETE Bucket request

GET(req)¶

Handle GET Bucket (List Objects) request

HEAD(req)¶

Handle HEAD Bucket (Get Metadata) request

POST(req)¶

Handle POST Bucket request

PUT(req)¶

Handle PUT Bucket request

class swift.common.middleware.s3api.controllers.obj.ObjectController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles requests on objects

DELETE(req)¶

Handle DELETE Object request

GET(req)¶

Handle GET Object request

HEAD(req)¶

Handle HEAD Object request

PUT(req)¶

Handle PUT Object and PUT Object (Copy) request

class swift.common.middleware.s3api.controllers.acl.AclController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles the following APIs:

• GET Bucket acl
• PUT Bucket acl
• GET Object acl
• PUT Object acl

Those APIs are logged as ACL operations in the S3 server log.

GET(req)¶

Handles GET Bucket acl and GET Object acl.

PUT(req)¶

Handles PUT Bucket acl and PUT Object acl.

swift.common.middleware.s3api.controllers.acl.get_acl(account_name, headers)¶

Attempts to construct an S3 ACL based on what is found in the swift headers

class swift.common.middleware.s3api.controllers.s3_acl.S3AclController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles the following APIs:

• GET Bucket acl
• PUT Bucket acl
• GET Object acl
• PUT Object acl

Those APIs are logged as ACL operations in the S3 server log.

GET(req)¶

Handles GET Bucket acl and GET Object acl.

PUT(req)¶

Handles PUT Bucket acl and PUT Object acl.

Implementation of S3 Multipart Upload.

This module implements S3 Multipart Upload APIs with the Swift SLO feature. The following explains how S3api uses swift container and objects to store S3 upload information:

[bucket]+segments/[upload_id1]  # upload id object for upload_id1
[bucket]+segments/[upload_id1]/1  # part object for upload_id1
[bucket]+segments/[upload_id1]/2  # part object for upload_id1
[bucket]+segments/[upload_id1]/3  # part object for upload_id1
[bucket]+segments/[upload_id2]  # upload id object for upload_id2
[bucket]+segments/[upload_id2]/1  # part object for upload_id2
[bucket]+segments/[upload_id2]/2  # part object for upload_id2
   .
   .

class swift.common.middleware.s3api.controllers.multi_delete.MultiObjectDeleteController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles Delete Multiple Objects, which is logged as a MULTI_OBJECT_DELETE operation in the S3 server log.

POST(req)¶

Handles Delete Multiple Objects.

class swift.common.middleware.s3api.controllers.versioning.VersioningController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles the following APIs:

• GET Bucket versioning
• PUT Bucket versioning

Those APIs are logged as VERSIONING operations in the S3 server log.

GET(req)¶

Handles GET Bucket versioning.

PUT(req)¶

Handles PUT Bucket versioning.

class swift.common.middleware.s3api.controllers.location.LocationController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles GET Bucket location, which is logged as a LOCATION operation in the S3 server log.

GET(req)¶

Handles GET Bucket location.

class swift.common.middleware.s3api.controllers.logging.LoggingStatusController(app, conf, logger, **kwargs)¶

Bases: swift.common.middleware.s3api.controllers.base.Controller

Handles the following APIs:

• GET Bucket logging
• PUT Bucket logging

Those APIs are logged as LOGGING_STATUS operations in the S3 server log.

GET(req)¶

Handles GET Bucket logging.

PUT(req)¶

Handles PUT Bucket logging.

Bulk Operations (Delete and Archive Auto Extraction)¶

Middleware that will perform many operations on a single request.

/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz

/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH

X-Detect-Content-Type: true

curl -X PUT http://127.0.0.1/v1/AUTH_acc/cont/$?extract-archive=tar
 -T backup.tar
 -H "Content-Type: application/x-tar"
 -H "X-Auth-Token: xxx"
 -H "X-Detect-Content-Type: true"

setfattr -n user.mime_type -v "application/python-setup" setup.py
setfattr -n user.meta.lunch -v "burger and fries" setup.py
setfattr -n user.meta.dinner -v "baked ziti" setup.py
setfattr -n user.stuff -v "whee" setup.py

Content-Type: application/python-setup
X-Object-Meta-Lunch: burger and fries
X-Object-Meta-Dinner: baked ziti

{"Response Status": "201 Created",
 "Response Body": "",
 "Errors": [],
 "Number Files Created": 10}

/container_name/obj_name

/container_name

{"Number Not Found": 0,
 "Response Status": "200 OK",
 "Response Body": "",
 "Errors": [],
 "Number Deleted": 6}

CatchErrors¶

exception swift.common.middleware.catch_errors.BadResponseLength¶

Bases: Exception

class swift.common.middleware.catch_errors.CatchErrorMiddleware(app, conf)¶

Bases: object

Middleware that provides high-level error handling and ensures that a transaction id will be set for every request.

class swift.common.middleware.catch_errors.CatchErrorsContext(app, logger, trans_id_suffix='')¶

Bases: swift.common.wsgi.WSGIContext

swift.common.middleware.catch_errors.enforce_byte_count(inner_iter, nbytes)¶

Enforces that inner_iter yields exactly <nbytes> bytes before exhaustion.

If inner_iter fails to do so, BadResponseLength is raised.

Parameters:	inner_iter – iterable of bytestrings nbytes – number of bytes expected

CNAME Lookup¶

CNAME Lookup Middleware

Middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain’s CNAME record in DNS.

This middleware will continue to follow a CNAME chain in DNS until it finds a record ending in the configured storage domain or it reaches the configured maximum lookup depth. If a match is found, the environment’s Host header is rewritten and the request is passed further down the WSGI chain.

class swift.common.middleware.cname_lookup.CNAMELookupMiddleware(app, conf)¶

Bases: object

CNAME Lookup Middleware

See above for a full description.

Parameters:	app – The next WSGI filter or app in the paste.deploy chain. conf – The configuration dict for the middleware.

swift.common.middleware.cname_lookup.lookup_cname(domain, resolver)¶

Given a domain, returns its DNS CNAME mapping and DNS ttl.

Parameters:	domain – domain to query on resolver – dns.resolver.Resolver() instance used for executing DNS queries
Returns:	(ttl, result)

Container Quotas¶

The container_quotas middleware implements simple quotas that can be imposed on swift containers by a user with the ability to set container metadata, most likely the account administrator. This can be useful for limiting the scope of containers that are delegated to non-admin users, exposed to formpost uploads, or just as a self-imposed sanity check.

Any object PUT operations that exceed these quotas return a 413 response (request entity too large) with a descriptive body.

Quotas are subject to several limitations: eventual consistency, the timeliness of the cached container_info (60 second ttl by default), and it’s unable to reject chunked transfer uploads that exceed the quota (though once the quota is exceeded, new chunked transfers will be refused).

Quotas are set by adding meta values to the container, and are validated when set:

The container_quotas middleware should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. For example:

[pipeline:main]
pipeline = catch_errors cache tempauth container_quotas proxy-server

[filter:container_quotas]
use = egg:swift#container_quotas

Container Sync Middleware¶

class swift.common.middleware.container_sync.ContainerSync(app, conf, logger=None)¶

Bases: object

WSGI middleware that validates an incoming container sync request using the container-sync-realms.conf style of container sync.

Cross Domain Policies¶

class swift.common.middleware.crossdomain.CrossDomainMiddleware(app, conf, *args, **kwargs)¶

Bases: object

Cross domain middleware used to respond to requests for cross domain policy information.

If the path is /crossdomain.xml it will respond with an xml cross domain policy document. This allows web pages hosted elsewhere to use client side technologies such as Flash, Java and Silverlight to interact with the Swift API.

To enable this middleware, add it to the pipeline in your proxy-server.conf file. It should be added before any authentication (e.g., tempauth or keystone) middleware. In this example ellipsis (…) indicate other middleware you may have chosen to use:

[pipeline:main]
pipeline =  ... crossdomain ... authtoken ... proxy-server

And add a filter section, such as:

[filter:crossdomain]
use = egg:swift#crossdomain
cross_domain_policy = <allow-access-from domain="*.example.com" />
    <allow-access-from domain="www.example.com" secure="false" />

For continuation lines, put some whitespace before the continuation text. Ensure you put a completely blank line to terminate the cross_domain_policy value.

The cross_domain_policy name/value is optional. If omitted, the policy defaults as if you had specified:

cross_domain_policy = <allow-access-from domain="*" secure="false" />

GET(req)¶

Returns a 200 response with cross domain policy information

Discoverability¶

Swift will by default provide clients with an interface providing details about the installation. Unless disabled (i.e expose_info=false in Proxy Server Configuration), a GET request to /info will return configuration data in JSON format. An example response:

{"swift": {"version": "1.11.0"}, "staticweb": {}, "tempurl": {}}

This would signify to the client that swift version 1.11.0 is running and that staticweb and tempurl are available in this installation.

There may be administrator-only information available via /info. To retrieve it, one must use an HMAC-signed request, similar to TempURL. The signature may be produced like so:

swift tempurl GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g

Domain Remap¶

Domain Remap Middleware

Middleware that translates container and account parts of a domain to path parameters that the proxy server understands.

Translation is only performed when the request URL’s host domain matches one of a list of domains. This list may be configured by the option storage_domain, and defaults to the single domain example.com.

If not already present, a configurable path_root, which defaults to v1, will be added to the start of the translated path.

For example, with the default configuration:

container.AUTH-account.example.com/object
container.AUTH-account.example.com/v1/object

would both be translated to:

container.AUTH-account.example.com/v1/AUTH_account/container/object

and:

AUTH-account.example.com/container/object
AUTH-account.example.com/v1/container/object

would both be translated to:

AUTH-account.example.com/v1/AUTH_account/container/object

Additionally, translation is only performed when the account name in the translated path starts with a reseller prefix matching one of a list configured by the option reseller_prefixes, or when no match is found but a default_reseller_prefix has been configured.

The reseller_prefixes list defaults to the single prefix AUTH. The default_reseller_prefix is not configured by default.

Browsers can convert a host header to lowercase, so the middleware checks that the reseller prefix on the account name is the correct case. This is done by comparing the items in the reseller_prefixes config option to the found prefix. If they match except for case, the item from reseller_prefixes will be used instead of the found reseller prefix. The middleware will also replace any hyphen (‘-‘) in the account name with an underscore (‘_’).

For example, with the default configuration:

auth-account.example.com/container/object
AUTH-account.example.com/container/object
auth_account.example.com/container/object
AUTH_account.example.com/container/object

would all be translated to:

<unchanged>.example.com/v1/AUTH_account/container/object

When no match is found in reseller_prefixes, the default_reseller_prefix config option is used. When no default_reseller_prefix is configured, any request with an account prefix not in the reseller_prefixes list will be ignored by this middleware.

For example, with default_reseller_prefix = AUTH:

account.example.com/container/object

would be translated to:

account.example.com/v1/AUTH_account/container/object

Note that this middleware requires that container names and account names (except as described above) must be DNS-compatible. This means that the account name created in the system and the containers created by users cannot exceed 63 characters or have UTF-8 characters. These are restrictions over and above what Swift requires and are not explicitly checked. Simply put, this middleware will do a best-effort attempt to derive account and container names from elements in the domain name and put those derived values into the URL path (leaving the Host header unchanged).

Also note that using Container to Container Synchronization with remapped domain names is not advised. With Container to Container Synchronization, you should use the true storage end points as sync destinations.

class swift.common.middleware.domain_remap.DomainRemapMiddleware(app, conf)¶

Bases: object

Domain Remap Middleware

See above for a full description.

Parameters:	app – The next WSGI filter or app in the paste.deploy chain. conf – The configuration dict for the middleware.

Dynamic Large Objects¶

DLO support centers around a user specified filter that matches segments and concatenates them together in object listing order. Please see the DLO docs for Dynamic Large Objects further details.

Encryption¶

Encryption middleware should be deployed in conjunction with the Keymaster middleware.

Implements middleware for object encryption which comprises an instance of a Decrypter combined with an instance of an Encrypter.

swift.common.middleware.crypto.filter_factory(global_conf, **local_conf)¶

Provides a factory function for loading encryption middleware.

class swift.common.middleware.crypto.encrypter.EncInputWrapper(crypto, keys, req, logger)¶

Bases: object

File-like object to be swapped in for wsgi.input.

class swift.common.middleware.crypto.encrypter.Encrypter(app, conf)¶

Bases: object

Middleware for encrypting data and user metadata.

By default all PUT or POST’ed object data and/or metadata will be encrypted. Encryption of new data and/or metadata may be disabled by setting the disable_encryption option to True. However, this middleware should remain in the pipeline in order for existing encrypted data to be read.

class swift.common.middleware.crypto.encrypter.EncrypterObjContext(encrypter, logger)¶

Bases: swift.common.middleware.crypto.crypto_utils.CryptoWSGIContext

encrypt_user_metadata(req, keys)¶

Encrypt user-metadata header values. Replace each x-object-meta-<key> user metadata header with a corresponding x-object-transient-sysmeta-crypto-meta-<key> header which has the crypto metadata required to decrypt appended to the encrypted value.

Parameters:	req – a swob Request keys – a dict of encryption keys

handle_post(req, start_response)¶

Encrypt the new object headers with a new iv and the current crypto. Note that an object may have encrypted headers while the body may remain unencrypted.

swift.common.middleware.crypto.encrypter.encrypt_header_val(crypto, value, key)¶

Encrypt a header value using the supplied key.

Parameters:	crypto – a Crypto instance value – value to encrypt key – crypto key to use
Returns:	a tuple of (encrypted value, crypto_meta) where crypto_meta is a dict of form returned by get_crypto_meta()
Raises:	ValueError – if value is empty

class swift.common.middleware.crypto.decrypter.BaseDecrypterContext(crypto_app, server_type, logger)¶

Bases: swift.common.middleware.crypto.crypto_utils.CryptoWSGIContext

decrypt_value(value, key, crypto_meta)¶

Base64-decode and decrypt a value using the crypto_meta provided.

Parameters:	value – a base64-encoded value to decrypt key – crypto key to use crypto_meta – a crypto-meta dict of form returned by get_crypto_meta()
Returns:	decrypted value

decrypt_value_with_meta(value, key, required=False)¶

Base64-decode and decrypt a value if crypto meta can be extracted from the value itself, otherwise return the value unmodified.

A value should either be a string that does not contain the ‘;’ character or should be of the form:

<base64-encoded ciphertext>;swift_meta=<crypto meta>

Parameters:	value – value to decrypt key – crypto key to use required – if True then the value is required to be decrypted and an EncryptionException will be raised if the header cannot be decrypted due to missing crypto meta.
Returns:	decrypted value if crypto meta is found, otherwise the unmodified value
Raises:	EncryptionException – if an error occurs while parsing crypto meta or if the header value was required to be decrypted but crypto meta was not found.

get_crypto_meta(header_name, check=True)¶

Extract a crypto_meta dict from a header.

Parameters:	header_name – name of header that may have crypto_meta check – if True validate the crypto meta
Returns:	A dict containing crypto_meta items
Raises:	EncryptionException – if an error occurs while parsing the crypto meta

get_decryption_keys(req, crypto_meta=None)¶

Determine if a response should be decrypted, and if so then fetch keys.

Parameters:	req – a Request object crypto_meta – a dict of crypto metadata
Returns:	a dict of decryption keys

get_unwrapped_key(crypto_meta, wrapping_key)¶

Get a wrapped key from crypto-meta and unwrap it using the provided wrapping key.

Parameters:	crypto_meta – a dict of crypto-meta wrapping_key – key to be used to decrypt the wrapped key
Returns:	an unwrapped key
Raises:	HTTPInternalServerError – if the crypto-meta has no wrapped key or the unwrapped key is invalid

class swift.common.middleware.crypto.decrypter.Decrypter(app, conf)¶

Bases: object

Middleware for decrypting data and user metadata.

class swift.common.middleware.crypto.decrypter.DecrypterContContext(decrypter, logger)¶

Bases: swift.common.middleware.crypto.decrypter.BaseDecrypterContext

process_json_resp(req, resp_iter)¶

Parses json body listing and decrypt encrypted entries. Updates Content-Length header with new body length and return a body iter.

class swift.common.middleware.crypto.decrypter.DecrypterObjContext(decrypter, logger)¶

Bases: swift.common.middleware.crypto.decrypter.BaseDecrypterContext

decrypt_resp_headers(put_keys, post_keys)¶

Find encrypted headers and replace with the decrypted versions.

Parameters:	put_keys – a dict of decryption keys used for object PUT. post_keys – a dict of decryption keys used for object POST.
Returns:	A list of headers with any encrypted headers replaced by their decrypted values.
Raises:	HTTPInternalServerError – if any error occurs while decrypting headers

multipart_response_iter(resp, boundary, body_key, crypto_meta)¶

Decrypts a multipart mime doc response body.

Parameters:	resp – application response boundary – multipart boundary string body_key – decryption key for the response body crypto_meta – crypto_meta for the response body
Returns:	generator for decrypted response body

response_iter(resp, body_key, crypto_meta, offset)¶

Decrypts a response body.

Parameters:	resp – application response body_key – decryption key for the response body crypto_meta – crypto_meta for the response body offset – offset into object content at which response body starts
Returns:	generator for decrypted response body

FormPost¶

FormPost Middleware

Translates a browser form post into a regular Swift object PUT.

The format of the form is:

<form action="<swift-url>" method="POST"
      enctype="multipart/form-data">
  <input type="hidden" name="redirect" value="<redirect-url>" />
  <input type="hidden" name="max_file_size" value="<bytes>" />
  <input type="hidden" name="max_file_count" value="<count>" />
  <input type="hidden" name="expires" value="<unix-timestamp>" />
  <input type="hidden" name="signature" value="<hmac>" />
  <input type="file" name="file1" /><br />
  <input type="submit" />
</form>

Optionally, if you want the uploaded files to be temporary you can set x-delete-at or x-delete-after attributes by adding one of these as a form input:

<input type="hidden" name="x_delete_at" value="<unix-timestamp>" />
<input type="hidden" name="x_delete_after" value="<seconds>" />

The <swift-url> is the URL of the Swift destination, such as:

https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix

The name of each file uploaded will be appended to the <swift-url> given. So, you can upload directly to the root of container with a url like:

https://swift-cluster.example.com/v1/AUTH_account/container/

Optionally, you can include an object prefix to better separate different users’ uploads, such as:

https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix

Note the form method must be POST and the enctype must be set as “multipart/form-data”.

The redirect attribute is the URL to redirect the browser to after the upload completes. This is an optional parameter. If you are uploading the form via an XMLHttpRequest the redirect should not be included. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”).

The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes.

The max_file_count attribute must be included and indicates the maximum number of files that can be uploaded with the form. Include additional <input type="file" name="filexx" /> attributes if desired.

The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated.

The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature:

import hmac
from hashlib import sha1
from time import time
path = '/v1/account/container/object_prefix'
redirect = 'https://srv.com/some-page'  # set to '' if redirect not in form
max_file_size = 104857600
max_file_count = 10
expires = int(time() + 600)
key = 'mykey'
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
    max_file_size, max_file_count, expires)
signature = hmac.new(key, hmac_body, sha1).hexdigest()

The key is the value of either the account (X-Account-Meta-Temp-URL-Key, X-Account-Meta-Temp-Url-Key-2) or the container (X-Container-Meta-Temp-URL-Key, X-Container-Meta-Temp-Url-Key-2) TempURL keys.

Be certain to use the full path, from the /v1/ onward. Note that x_delete_at and x_delete_after are not used in signature generation as they are both optional attributes.

The command line tool swift-form-signature may be used (mostly just when testing) to compute expires and signature.

Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).

class swift.common.middleware.formpost.FormPost(app, conf)¶

Bases: object

FormPost Middleware

See above for a full description.

The proxy logs created for any subrequests made will have swift.source set to “FP”.

Parameters:	app – The next WSGI filter or app in the paste.deploy chain. conf – The configuration dict for the middleware.

app = None¶

The next WSGI application/filter in the paste.deploy pipeline.

conf = None¶

The filter configuration dict.

swift.common.middleware.formpost.filter_factory(global_conf, **local_conf)¶

Returns the WSGI filter for use with paste.deploy.

swift.common.middleware.formpost.READ_CHUNK_SIZE = 4096¶

The size of data to read from the form at any given time.

swift.common.middleware.formpost.MAX_VALUE_LENGTH = 4096¶

The maximum size of any attribute’s value. Any additional data will be truncated.

GateKeeper¶

The gatekeeper middleware imposes restrictions on the headers that may be included with requests and responses. Request headers are filtered to remove headers that should never be generated by a client. Similarly, response headers are filtered to remove private headers that should never be passed to a client.

The gatekeeper middleware must always be present in the proxy server wsgi pipeline. It should be configured close to the start of the pipeline specified in /etc/swift/proxy-server.conf, immediately after catch_errors and before any other middleware. It is essential that it is configured ahead of all middlewares using system metadata in order that they function correctly.

If gatekeeper middleware is not configured in the pipeline then it will be automatically inserted close to the start of the pipeline by the proxy server.

swift.common.middleware.gatekeeper.outbound_exclusions = ['x-account-sysmeta-', 'x-container-sysmeta-', 'x-object-sysmeta-', 'x-object-transient-sysmeta-', 'x-backend']¶

A list of python regular expressions that will be used to match against outbound response headers. Matching headers will be removed from the response.

Healthcheck¶

class swift.common.middleware.healthcheck.HealthCheckMiddleware(app, conf)¶

Bases: object

Healthcheck middleware used for monitoring.

If the path is /healthcheck, it will respond 200 with “OK” as the body.

If the optional config parameter “disable_path” is set, and a file is present at that path, it will respond 503 with “DISABLED BY FILE” as the body.

DISABLED(req)¶

Returns a 503 response with “DISABLED BY FILE” in the body.

GET(req)¶

Returns a 200 response with “OK” in the body.

Keymaster¶

Keymaster middleware should be deployed in conjunction with the Encryption middleware.

class swift.common.middleware.crypto.keymaster.KeyMaster(app, conf)¶

Bases: object

Middleware for providing encryption keys.

The middleware requires at least one encryption root secret(s) to be set. This is the root secret from which encryption keys are derived. This must be set before first use to a value that is at least 256 bits. The security of all encrypted data critically depends on this key, therefore it should be set to a high-entropy value. For example, a suitable value may be obtained by generating a 32 byte (or longer) value using a cryptographically secure random number generator. Changing the root secret is likely to result in data loss.

create_key(path, secret_id=None)¶

Creates an encryption key that is unique for the given path.

Parameters:	path – the path of the resource being encrypted. secret_id – the id of the root secret from which the key should be derived.
Returns:	an encryption key.
Raises:	UnknownSecretIdError – if the secret_id is not recognised.

class swift.common.middleware.crypto.keymaster.KeyMasterContext(keymaster, account, container, obj, meta_version_to_write='2')¶

Bases: swift.common.wsgi.WSGIContext

The simple scheme for key derivation is as follows: every path is associated with a key, where the key is derived from the path itself in a deterministic fashion such that the key does not need to be stored. Specifically, the key for any path is an HMAC of a root key and the path itself, calculated using an SHA256 hash function:

<path_key> = HMAC_SHA256(<root_secret>, <path>)

fetch_crypto_keys(key_id=None, *args, **kwargs)¶

Setup container and object keys based on the request path.

Keys are derived from request path. The ‘id’ entry in the results dict includes the part of the path used to derive keys. Other keymaster implementations may use a different strategy to generate keys and may include a different type of ‘id’, so callers should treat the ‘id’ as opaque keymaster-specific data.

Parameters:	key_id – if given this should be a dict with the items included under the id key of a dict returned by this method.
Returns:	A dict containing encryption keys for ‘object’ and ‘container’, and entries ‘id’ and ‘all_ids’. The ‘all_ids’ entry is a list of key id dicts for all root secret ids including the one used to generate the returned keys.

KeystoneAuth¶

class swift.common.middleware.keystoneauth.KeystoneAuth(app, conf)¶

Bases: object

Swift middleware to Keystone authorization system.

In Swift’s proxy-server.conf add this keystoneauth middleware and the authtoken middleware to your pipeline. Make sure you have the authtoken middleware before the keystoneauth middleware.

The authtoken middleware will take care of validating the user and keystoneauth will authorize access.

The sample proxy-server.conf shows a sample pipeline that uses keystone.

proxy-server.conf-sample

The authtoken middleware is shipped with keystonemiddleware - it does not have any other dependencies than itself so you can either install it by copying the file directly in your python path or by installing keystonemiddleware.

If support is required for unvalidated users (as with anonymous access) or for formpost/staticweb/tempurl middleware, authtoken will need to be configured with delay_auth_decision set to true. See the Keystone documentation for more detail on how to configure the authtoken middleware.

In proxy-server.conf you will need to have the setting account auto creation to true:

[app:proxy-server]
account_autocreate = true

And add a swift authorization filter section, such as:

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin, swiftoperator

The user who is able to give ACL / create Containers permissions will be the user with a role listed in the operator_roles setting which by default includes the admin and the swiftoperator roles.

The keystoneauth middleware maps a Keystone project/tenant to an account in Swift by adding a prefix (AUTH_ by default) to the tenant/project id.. For example, if the project id is 1234, the path is /v1/AUTH_1234.

If you need to have a different reseller_prefix to be able to mix different auth servers you can configure the option reseller_prefix in your keystoneauth entry like this:

reseller_prefix = NEWAUTH

Don’t forget to also update the Keystone service endpoint configuration to use NEWAUTH in the path.

It is possible to have several accounts associated with the same project. This is done by listing several prefixes as shown in the following example:

reseller_prefix = AUTH, SERVICE

This means that for project id ‘1234’, the paths ‘/v1/AUTH_1234’ and ‘/v1/SERVICE_1234’ are associated with the project and are authorized using roles that a user has with that project. The core use of this feature is that it is possible to provide different rules for each account prefix. The following parameters may be prefixed with the appropriate prefix:

operator_roles
service_roles

For backward compatibility, if either of these parameters is specified without a prefix then it applies to all reseller_prefixes. Here is an example, using two prefixes:

reseller_prefix = AUTH, SERVICE
# The next three lines have identical effects (since the first applies
# to both prefixes).
operator_roles = admin, swiftoperator
AUTH_operator_roles = admin, swiftoperator
SERVICE_operator_roles = admin, swiftoperator
# The next line only applies to accounts with the SERVICE prefix
SERVICE_operator_roles = admin, some_other_role

X-Service-Token tokens are supported by the inclusion of the service_roles configuration option. When present, this option requires that the X-Service-Token header supply a token from a user who has a role listed in service_roles. Here is an example configuration:

reseller_prefix = AUTH, SERVICE
AUTH_operator_roles = admin, swiftoperator
SERVICE_operator_roles = admin, swiftoperator
SERVICE_service_roles = service

The keystoneauth middleware supports cross-tenant access control using the syntax <tenant>:<user> to specify a grantee in container Access Control Lists (ACLs). For a request to be granted by an ACL, the grantee <tenant> must match the UUID of the tenant to which the request X-Auth-Token is scoped and the grantee <user> must match the UUID of the user authenticated by that token.

Note that names must no longer be used in cross-tenant ACLs because with the introduction of domains in keystone names are no longer globally unique.

For backwards compatibility, ACLs using names will be granted by keystoneauth when it can be established that the grantee tenant, the grantee user and the tenant being accessed are either not yet in a domain (e.g. the X-Auth-Token has been obtained via the keystone v2 API) or are all in the default domain to which legacy accounts would have been migrated. The default domain is identified by its UUID, which by default has the value default. This can be changed by setting the default_domain_id option in the keystoneauth configuration:

default_domain_id = default

The backwards compatible behavior can be disabled by setting the config option allow_names_in_acls to false:

allow_names_in_acls = false

To enable this backwards compatibility, keystoneauth will attempt to determine the domain id of a tenant when any new account is created, and persist this as account metadata. If an account is created for a tenant using a token with reselleradmin role that is not scoped on that tenant, keystoneauth is unable to determine the domain id of the tenant; keystoneauth will assume that the tenant may not be in the default domain and therefore not match names in ACLs for that account.

By default, middleware higher in the WSGI pipeline may override auth processing, useful for middleware such as tempurl and formpost. If you know you’re not going to use such middleware and you want a bit of extra security you can disable this behaviour by setting the allow_overrides option to false:

allow_overrides = false

Parameters:	app – The next WSGI app in the pipeline conf – The dict of configuration values

authorize_anonymous(req)¶

Authorize an anonymous request.

Returns:	None if authorization is granted, an error page otherwise.

denied_response(req)¶

Deny WSGI Response.

Returns a standard WSGI response callable with the status of 403 or 401 depending on whether the REMOTE_USER is set or not.

swift.common.middleware.keystoneauth.filter_factory(global_conf, **local_conf)¶

Returns a WSGI filter app for use with paste.deploy.

List Endpoints¶

List endpoints for an object, account or container.

This middleware makes it possible to integrate swift with software that relies on data locality information to avoid network overhead, such as Hadoop.

Using the original API, answers requests of the form:

/endpoints/{account}/{container}/{object}
/endpoints/{account}/{container}
/endpoints/{account}
/endpoints/v1/{account}/{container}/{object}
/endpoints/v1/{account}/{container}
/endpoints/v1/{account}

with a JSON-encoded list of endpoints of the form:

http://{server}:{port}/{dev}/{part}/{acc}/{cont}/{obj}
http://{server}:{port}/{dev}/{part}/{acc}/{cont}
http://{server}:{port}/{dev}/{part}/{acc}

correspondingly, e.g.:

http://10.1.1.1:6200/sda1/2/a/c2/o1
http://10.1.1.1:6200/sda1/2/a/c2
http://10.1.1.1:6200/sda1/2/a

Using the v2 API, answers requests of the form:

/endpoints/v2/{account}/{container}/{object}
/endpoints/v2/{account}/{container}
/endpoints/v2/{account}

with a JSON-encoded dictionary containing a key ‘endpoints’ that maps to a list of endpoints having the same form as described above, and a key ‘headers’ that maps to a dictionary of headers that should be sent with a request made to the endpoints, e.g.:

{ "endpoints": {"http://10.1.1.1:6010/sda1/2/a/c3/o1",
                "http://10.1.1.1:6030/sda3/2/a/c3/o1",
                "http://10.1.1.1:6040/sda4/2/a/c3/o1"},
  "headers": {"X-Backend-Storage-Policy-Index": "1"}}

In this example, the ‘headers’ dictionary indicates that requests to the endpoint URLs should include the header ‘X-Backend-Storage-Policy-Index: 1’ because the object’s container is using storage policy index 1.

The ‘/endpoints/’ path is customizable (‘list_endpoints_path’ configuration parameter).

Intended for consumption by third-party services living inside the cluster (as the endpoints make sense only inside the cluster behind the firewall); potentially written in a different language.

This is why it’s provided as a REST API and not just a Python API: to avoid requiring clients to write their own ring parsers in their languages, and to avoid the necessity to distribute the ring file to clients and keep it up-to-date.

Note that the call is not authenticated, which means that a proxy with this middleware enabled should not be open to an untrusted environment (everyone can query the locality data using this middleware).

class swift.common.middleware.list_endpoints.ListEndpointsMiddleware(app, conf)¶

Bases: object

List endpoints for an object, account or container.

See above for a full description.

Uses configuration parameter swift_dir (default /etc/swift).

Parameters:	app – The next WSGI filter or app in the paste.deploy chain. conf – The configuration dict for the middleware.

get_object_ring(policy_idx)¶

Get the ring object to use to handle a request based on its policy.

Policy_idx:	policy index as defined in swift.conf
Returns:	appropriate ring object

Memcache¶

class swift.common.middleware.memcache.MemcacheMiddleware(app, conf)¶

Bases: object

Caching middleware that manages caching in swift.

Name Check (Forbidden Character Filter)¶

Created on February 27, 2012

A filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.

Place early in the proxy-server pipeline after the left-most occurrence of the proxy-logging middleware (if present) and before the final proxy-logging middleware (if present) or the proxy-serer app itself, e.g.:

[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging name_check cache ratelimit tempauth sos proxy-logging proxy-server

[filter:name_check]
use = egg:swift#name_check
forbidden_chars = '"`<>
maximum_length = 255

There are default settings for forbidden_chars (FORBIDDEN_CHARS) and maximum_length (MAX_LENGTH)

The filter returns HTTPBadRequest if path is invalid.

@author: eamonn-otoole

Object Versioning¶

Object versioning in swift is implemented by setting a flag on the container to tell swift to version all objects in the container. The value of the flag is the URL-encoded container name where the versions are stored (commonly referred to as the “archive container”). The flag itself is one of two headers, which determines how object DELETE requests are handled:

• X-History-Location

  On DELETE, copy the current version of the object to the archive container, write a zero-byte “delete marker” object that notes when the delete took place, and delete the object from the versioned container. The object will no longer appear in container listings for the versioned container and future requests there will return 404 Not Found. However, the content will still be recoverable from the archive container.

• X-Versions-Location

  On DELETE, only remove the current version of the object. If any previous versions exist in the archive container, the most recent one is copied over the current version, and the copy in the archive container is deleted. As a result, if you have 5 total versions of the object, you must delete the object 5 times for that object name to start responding with 404 Not Found.

Either header may be used for the various containers within an account, but only one may be set for any given container. Attempting to set both simulataneously will result in a 400 Bad Request response.

When data is PUT into a versioned container (a container with the versioning flag turned on), the existing data in the file is redirected to a new object in the archive container and the data in the PUT request is saved as the data for the versioned object. The new object name (for the previous version) is <archive_container>/<length><object_name>/<timestamp>, where length is the 3-character zero-padded hexadecimal length of the <object_name> and <timestamp> is the timestamp of when the previous version was created.

A GET to a versioned object will return the current version of the object without having to do any request redirects or metadata lookups.

A POST to a versioned object will update the object metadata as normal, but will not create a new version of the object. In other words, new versions are only created when the content of the object changes.

A DELETE to a versioned object will be handled in one of two ways, as described above.

To restore a previous version of an object, find the desired version in the archive container then issue a COPY with a Destination header indicating the original location. This will archive the current version similar to a PUT over the versioned object. If the client additionally wishes to permanently delete what was the current version, it must find the newly-created archive in the archive container and issue a separate DELETE to it.

curl -i -XPUT -H "X-Auth-Token: <token>" -H "X-Versions-Location: versions" http://<storage_url>/container
curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions

curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/

curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/
curl -i -XGET -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -XPUT -H "X-Auth-Token: <token>" -H "X-History-Location: versions" http://<storage_url>/container
curl -i -XPUT -H "X-Auth-Token: <token>" http://<storage_url>/versions

curl -i -XPUT --data-binary 1 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -XPUT --data-binary 2 -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject
curl -i -H "X-Auth-Token: <token>" http://<storage_url>/container/myobject

curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/

curl -i -XCOPY -H "X-Auth-Token: <token>" http://<storage_url>/versions/008myobject/<timestamp> -H "Destination: container/myobject"

curl -i -H "X-Auth-Token: <token>" http://<storage_url>/versions?prefix=008myobject/

curl -i -XDELETE -H "X-Auth-Token: <token>" http://<storage_url>/versions/008myobject/<timestamp>

curl -i -XPOST -H "X-Auth-Token: <token>" -H "X-Remove-Versions-Location: x" http://<storage_url>/container

Proxy Logging¶

Logging middleware for the Swift proxy.

This serves as both the default logging implementation and an example of how to plug in your own logging format/method.

The logging format implemented below is as follows:

client_ip remote_addr datetime request_method request_path protocol

status_int referer user_agent auth_token bytes_recvd bytes_sent client_etag transaction_id headers request_time source log_info request_start_time request_end_time

These values are space-separated, and each is url-encoded, so that they can be separated with a simple .split()

• remote_addr is the contents of the REMOTE_ADDR environment variable, while client_ip is swift’s best guess at the end-user IP, extracted variously from the X-Forwarded-For header, X-Cluster-Ip header, or the REMOTE_ADDR environment variable.
• source (swift.source in the WSGI environment) indicates the code that generated the request, such as most middleware. (See below for more detail.)
• log_info (swift.log_info in the WSGI environment) is for additional information that could prove quite useful, such as any x-delete-at value or other “behind the scenes” activity that might not otherwise be detectable from the plain log information. Code that wishes to add additional log information should use code like env.setdefault('swift.log_info', []).append(your_info) so as to not disturb others’ log information.
• Values that are missing (e.g. due to a header not being present) or zero are generally represented by a single hyphen (‘-‘).

The proxy-logging can be used twice in the proxy server’s pipeline when there is middleware installed that can return custom responses that don’t follow the standard pipeline to the proxy server.

For example, with staticweb, the middleware might intercept a request to /v1/AUTH_acc/cont/, make a subrequest to the proxy to retrieve /v1/AUTH_acc/cont/index.html and, in effect, respond to the client’s original request using the 2nd request’s body. In this instance the subrequest will be logged by the rightmost middleware (with a swift.source set) and the outgoing request (with body overridden) will be logged by leftmost middleware.

Requests that follow the normal pipeline (use the same wsgi environment throughout) will not be double logged because an environment variable (swift.proxy_access_log_made) is checked/set when a log is made.

All middleware making subrequests should take care to set swift.source when needed. With the doubled proxy logs, any consumer/processor of swift’s proxy logs should look at the swift.source field, the rightmost log value, to decide if this is a middleware subrequest or not. A log processor calculating bandwidth usage will want to only sum up logs with no swift.source.

class swift.common.middleware.proxy_logging.ProxyLoggingMiddleware(app, conf, logger=None)¶

Bases: object

Middleware that logs Swift proxy requests in the swift log format.

log_request(req, status_int, bytes_received, bytes_sent, start_time, end_time, resp_headers=None)¶

Log a request.

Parameters:	req – swob.Request object for the request status_int – integer code for the response status bytes_received – bytes successfully read from the request body bytes_sent – bytes yielded to the WSGI server start_time – timestamp request started end_time – timestamp request completed resp_headers – dict of the response headers

Ratelimit¶

exception swift.common.middleware.ratelimit.MaxSleepTimeHitError¶

Bases: Exception

class swift.common.middleware.ratelimit.RateLimitMiddleware(app, conf, logger=None)¶

Bases: object

Rate limiting middleware

Rate limits requests on both an Account and Container level. Limits are configurable.

get_ratelimitable_key_tuples(req, account_name, container_name=None, obj_name=None, global_ratelimit=None)¶

Returns a list of key (used in memcache), ratelimit tuples. Keys should be checked in order.

Parameters:	req – swob request account_name – account name from path container_name – container name from path obj_name – object name from path global_ratelimit – this account has an account wide ratelimit on all writes combined

handle_ratelimit(req, account_name, container_name, obj_name)¶

Performs rate limiting and account white/black listing. Sleeps if necessary. If self.memcache_client is not set, immediately returns None.

Parameters:	account_name – account name from path container_name – container name from path obj_name – object name from path

swift.common.middleware.ratelimit.filter_factory(global_conf, **local_conf)¶

paste.deploy app factory for creating WSGI proxy apps.

swift.common.middleware.ratelimit.get_maxrate(ratelimits, size)¶

Returns number of requests allowed per second for given size.

swift.common.middleware.ratelimit.interpret_conf_limits(conf, name_prefix, info=None)¶

Parses general parms for rate limits looking for things that start with the provided name_prefix within the provided conf and returns lists for both internal use and for /info

Parameters:	conf – conf dict to parse name_prefix – prefix of config parms to look for info – set to return extra stuff for /info registration

Read Only¶

class swift.common.middleware.read_only.ReadOnlyMiddleware(app, conf, logger=None)¶

Bases: object

Middleware that make an entire cluster or individual accounts read only.

account_read_only(req, account)¶

Check whether an account should be read-only.

This considers both the cluster-wide config value as well as the per-account override in X-Account-Sysmeta-Read-Only.

swift.common.middleware.read_only.filter_factory(global_conf, **local_conf)¶

paste.deploy app factory for creating WSGI proxy apps.

Recon¶

class swift.common.middleware.recon.ReconMiddleware(app, conf, *args, **kwargs)¶

Bases: object

Recon middleware used for monitoring.

/recon/load|mem|async… will return various system metrics.

Needs to be added to the pipeline and requires a filter declaration in the object-server.conf:

[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift

get_async_info()¶

get # of async pendings

get_auditor_info(recon_type)¶

get auditor info

get_device_info()¶

get devices

get_diskusage()¶

get disk utilization statistics

get_driveaudit_error()¶

get # of drive audit errors

get_expirer_info(recon_type)¶

get expirer info

get_load(openr=<built-in function open>)¶

get info from /proc/loadavg

get_mem(openr=<built-in function open>)¶

get info from /proc/meminfo

get_mounted(openr=<built-in function open>)¶

get ALL mounted fs from /proc/mounts

get_quarantine_count()¶

get obj/container/account quarantine counts

get_replication_info(recon_type)¶

get replication info

get_ring_md5()¶

get all ring md5sum’s

get_socket_info(openr=<built-in function open>)¶

get info from /proc/net/sockstat and sockstat6

Note: The mem value is actually kernel pages, but we return bytes allocated based on the systems page size.

get_swift_conf_md5()¶

get md5 of swift.conf

get_time()¶

get current time

get_unmounted()¶

list unmounted (failed?) devices

get_updater_info(recon_type)¶

get updater info

get_version()¶

get swift version

Server Side Copy¶

Server side copy is a feature that enables users/clients to COPY objects between accounts and containers without the need to download and then re-upload objects, thus eliminating additional bandwidth consumption and also saving time. This may be used when renaming/moving an object which in Swift is a (COPY + DELETE) operation.

The server side copy middleware should be inserted in the pipeline after auth and before the quotas and large object middlewares. If it is not present in the pipeline in the proxy-server configuration file, it will be inserted automatically. There is no configurable option provided to turn off server side copy.

?multipart-manifest=get

Static Large Objects¶

Please see the SLO docs for Static Large Objects further details.

StaticWeb¶

This StaticWeb WSGI middleware will serve container data as a static web site with index file and error file resolution and optional file listings. This mode is normally only active for anonymous requests. When using keystone for authentication set delay_auth_decision = true in the authtoken middleware configuration in your /etc/swift/proxy-server.conf file. If you want to use it with authenticated requests, set the X-Web-Mode: true header on the request.

The staticweb filter should be added to the pipeline in your /etc/swift/proxy-server.conf file just after any auth middleware. Also, the configuration section for the staticweb middleware itself needs to be added. For example:

[DEFAULT]
...

[pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit tempauth
           staticweb proxy-logging proxy-server

...

[filter:staticweb]
use = egg:swift#staticweb

Any publicly readable containers (for example, X-Container-Read: .r:*, see ACLs for more information on this) will be checked for X-Container-Meta-Web-Index and X-Container-Meta-Web-Error header values:

X-Container-Meta-Web-Index  <index.name>
X-Container-Meta-Web-Error  <error.name.suffix>

If X-Container-Meta-Web-Index is set, any <index.name> files will be served without having to specify the <index.name> part. For instance, setting X-Container-Meta-Web-Index: index.html will be able to serve the object …/pseudo/path/index.html with just …/pseudo/path or …/pseudo/path/

If X-Container-Meta-Web-Error is set, any errors (currently just 401 Unauthorized and 404 Not Found) will instead serve the …/<status.code><error.name.suffix> object. For instance, setting X-Container-Meta-Web-Error: error.html will serve …/404error.html for requests for paths not found.

For pseudo paths that have no <index.name>, this middleware can serve HTML file listings if you set the X-Container-Meta-Web-Listings: true metadata item on the container.

If listings are enabled, the listings can have a custom style sheet by setting the X-Container-Meta-Web-Listings-CSS header. For instance, setting X-Container-Meta-Web-Listings-CSS: listing.css will make listings link to the …/listing.css style sheet. If you “view source” in your browser on a listing page, you will see the well defined document structure that can be styled.

By default, the listings will be rendered with a label of “Listing of /v1/account/container/path”. This can be altered by setting a X-Container-Meta-Web-Listings-Label: <label>. For example, if the label is set to “example.com”, a label of “Listing of example.com/path” will be used instead.

The content-type of directory marker objects can be modified by setting the X-Container-Meta-Web-Directory-Type header. If the header is not set, application/directory is used by default. Directory marker objects are 0-byte objects that represent directories to create a simulated hierarchical structure.

Example usage of this middleware via swift:

Make the container publicly readable:

swift post -r '.r:*' container

You should be able to get objects directly, but no index.html resolution or listings.

Set an index file directive:

swift post -m 'web-index:index.html' container

You should be able to hit paths that have an index.html without needing to type the index.html part.

Turn on listings:

swift post -r '.r:*,.rlistings' container
swift post -m 'web-listings: true' container

Now you should see object listings for paths and pseudo paths that have no index.html.

Enable a custom listings style sheet:

swift post -m 'web-listings-css:listings.css' container

Set an error file:

swift post -m 'web-error:error.html' container

Now 401’s should load 401error.html, 404’s should load 404error.html, etc.

Set Content-Type of directory marker object:

swift post -m 'web-directory-type:text/directory' container

Now 0-byte objects with a content-type of text/directory will be treated as directories rather than objects.

class swift.common.middleware.staticweb.StaticWeb(app, conf)¶

Bases: object

The Static Web WSGI middleware filter; serves container data as a static web site. See staticweb for an overview.

The proxy logs created for any subrequests made will have swift.source set to “SW”.

Parameters:	app – The next WSGI application/filter in the paste.deploy pipeline. conf – The filter configuration dict.

app = None¶

The next WSGI application/filter in the paste.deploy pipeline.

conf = None¶

The filter configuration dict. Only used in tests.

swift.common.middleware.staticweb.filter_factory(global_conf, **local_conf)¶

Returns a Static Web WSGI filter for use with paste.deploy.

Symlink¶

Symlink Middleware

Symlinks are objects stored in Swift that contain a reference to another object (hereinafter, this is called “target object”). They are analogous to symbolic links in Unix-like operating systems. The existence of a symlink object does not affect the target object in any way. An important use case is to use a path in one container to access an object in a different container, with a different policy. This allows policy cost/performance trade-offs to be made on individual objects.

Clients create a Swift symlink by performing a zero-length PUT request with the header X-Symlink-Target: <container>/<object>. For a cross-account symlink, the header X-Symlink-Target-Account: <account> must be included. If omitted, it is inserted automatically with the account of the symlink object in the PUT request process.

Symlinks must be zero-byte objects. Attempting to PUT a symlink with a non-empty request body will result in a 400-series error. Also, POST with X-Symlink-Target header always results in a 400-series error. The target object need not exist at symlink creation time. It is suggested to set the Content-Type of symlink objects to a distinct value such as application/symlink.

A GET/HEAD request to a symlink will result in a request to the target object referenced by the symlink’s X-Symlink-Target-Account and X-Symlink-Target headers. The response of the GET/HEAD request will contain a Content-Location header with the path location of the target object. A GET/HEAD request to a symlink with the query parameter ?symlink=get will result in the request targeting the symlink itself.

A symlink can point to another symlink. Chained symlinks will be traversed until target is not a symlink. If the number of chained symlinks exceeds the limit symloop_max an error response will be produced. The value of symloop_max can be defined in the symlink config section of proxy-server.conf. If not specified, the default symloop_max value is 2. If a value less than 1 is specified, the default value will be used.

A HEAD/GET request to a symlink object behaves as a normal HEAD/GET request to the target object. Therefore issuing a HEAD request to the symlink will return the target metadata, and issuing a GET request to the symlink will return the data and metadata of the target object. To return the symlink metadata (with its empty body) a GET/HEAD request with the ?symlink=get query parameter must be sent to a symlink object.

A POST request to a symlink will result in a 307 TemporaryRedirect response. The response will contain a Location header with the path of the target object as the value. The request is never redirected to the target object by Swift. Nevertheless, the metadata in the POST request will be applied to the symlink because object servers cannot know for sure if the current object is a symlink or not in eventual consistency.

A DELETE request to a symlink will delete the symlink itself. The target object will not be deleted.

A COPY request, or a PUT request with a X-Copy-From header, to a symlink will copy the target object. The same request to a symlink with the query parameter ?symlink=get will copy the symlink itself.

An OPTIONS request to a symlink will respond with the options for the symlink only, the request will not be redirected to the target object. Please note that if the symlink’s target object is in another container with CORS settings, the response will not reflect the settings.

Tempurls can be used to GET/HEAD symlink objects, but PUT is not allowed and will result in a 400-series error. The GET/HEAD tempurls honor the scope of the tempurl key. Container tempurl will only work on symlinks where the target container is the same as the symlink. In case a symlink targets an object in a different container, a GET/HEAD request will result in a 401 Unauthorized error. The account level tempurl will allow cross container symlinks.

If a symlink object is overwritten while it is in a versioned container, the symlink object itself is versioned, not the referenced object.

A GET request with query parameter ?format=json to a container which contains symlinks will respond with additional information symlink_path for each symlink object in the container listing. The symlink_path value is the target path of the symlink. Clients can differentiate symlinks and other objects by this function. Note that responses of any other format (e.g.``?format=xml``) won’t include symlink_path info.

Errors

• PUT with the header X-Symlink-Target with non-zero Content-Length will produce a 400 BadRequest error.
• POST with the header X-Symlink-Target will produce a 400 BadRequest error.
• GET/HEAD traversing more than symloop_max chained symlinks will produce a 409 Conflict error.
• POSTs will produce a 307 TemporaryRedirect error.