The zaqar.storage.mongodb.utils module

class HookedCursor(cursor, denormalizer)

Bases: object

next(*args, **kwargs)
cached_gen(iterable)

Converts the iterable into a caching generator.

Returns a proxy that yields each item of iterable, while at the same time caching those items in a deque.

Parameters:iterable – an iterable to wrap in a caching generator
Returns:(proxy(iterable), cached_items)
calculate_backoff(attempt, max_attempts, max_sleep, max_jitter=0)

Calculates backoff time, in seconds, when retrying an operation.

This function calculates a simple linear backoff time with optional jitter, useful for retrying a request under high concurrency.

The result may be passed directly into time.sleep() in order to mitigate stampeding herd syndrome and introduce backpressure towards the clients, slowing them down.

Parameters:
  • attempt – current value of the attempt counter (zero-based)
  • max_attempts – maximum number of attempts that will be tried
  • max_sleep – maximum sleep value to apply before jitter, assumed to be seconds. Fractional seconds are supported to 1 ms granularity.
  • max_jitter – maximum jitter value to add to the baseline sleep time. Actual value will be chosen randomly.
Raises:

ValueError
Returns:

float representing the number of seconds to sleep, within the interval [0, max_sleep), determined linearly according to the ratio attempt / max_attempts, with optional jitter.
descope_queue_name(scoped_name)

Returns the unscoped queue name, given a fully-scoped name.

get_partition(num_partitions, queue, project=None)

Get the partition number for a given queue and project.

Hashes the queue to a partition number. The hash is stable, meaning given the same queue name and project ID, the same partition number will always be returned. Note also that queues will be uniformly distributed across partitions.

The number of partitions is taken from the “partitions” property in the config file, under the [drivers:storage:mongodb] section.

normalize_none_str(string_or_none)

Returns ‘’ IFF given value is None, passthrough otherwise.

This function normalizes None to the empty string to facilitate string concatenation when a variable could be None.

oid_ts(oid)

Converts an ObjectId to a UNIX timestamp.

Raises:TypeError if oid isn’t an ObjectId
parse_scoped_project_queue(scoped_name)

Returns the project and queue name for a scoped catalogue entry.

Parameters:scoped_name (six.text_type) – a project/queue as given by :scope_queue_name:
Returns:(project, queue)
Return type:(six.text_type, six.text_type)
raises_conn_error(func)

Handles the MongoDB ConnectionFailure error.

This decorator catches MongoDB’s ConnectionFailure error and raises Zaqar’s ConnectionError instead.

retries_on_autoreconnect(func)

Causes the wrapped function to be re-called on AutoReconnect.

This decorator catches MongoDB’s AutoReconnect error and retries the function call.

Note

Assumes that the decorated function has defined self.driver.mongodb_conf so that max_reconnect_attempts and reconnect_sleep can be taken into account.

Warning

The decorated function must be idempotent.

scope_queue_name(queue=None, project=None)

Returns a scoped name for a queue based on project and queue.

If only the project name is specified, a scope signifying “all queues” for that project is returned. If neither queue nor project are specified, a scope for “all global queues” is returned, which is to be interpreted as excluding queues scoped by project.

Returns:‘{project}/{queue}’ if project and queue are given, ‘{project}/’ if ONLY project is given, ‘/{queue}’ if ONLY queue is given, and ‘/’ if neither are given.
scoped_query(queue, project)

Returns a dict usable for querying for scoped project/queues.

Parameters:
  • queue (six.text_type) – name of queue to seek
  • project (six.text_type) – namespace
  • key (six.text_type) – query key to use
Returns:

query to issue
Return type:

dict
stat_message(message, now)

Creates a stat document from the given message, relative to now.

to_oid(obj)

Creates a new ObjectId based on the input.

Returns None when TypeError or berrors.InvalidId is raised by the ObjectId class.

Parameters:obj – Anything that can be passed as an input to objectid.ObjectId