DynamORM API¶

`dynamorm`¶

The base module namespace simply imports the most frequently used objects to simplify imports in clients:

from dynamorm import DynaModel

`dynamorm.model`¶

Models represent tables in DynamoDB and define the characteristics of the Dynamo service as well as the Marshmallow or Schematics schema that is used for validating and marshalling your data.

class dynamorm.model.DynaModel(partial=False, **raw)¶

DynaModel is the base class all of your models will extend from. This model definition encapsulates the parameters used to create and manage the table as well as the schema for validating and marshalling data into object attributes. It will also hold any custom business logic you need for your objects.

Your class must define two inner classes that specify the Dynamo Table options and the Schema, respectively.

The Dynamo Table options are defined in a class named Table. See the dynamorm.table module for more information.

Any Local or Global Secondary Indexes you wish to create are defined as inner tables that extend from either the LocalIndex or GlobalIndex classes. See the dynamorm.table module for more information.

The document schema is defined in a class named Schema, which should be filled out exactly as you would fill out any other Marshmallow Schema or Schematics Model.

For example:

# Marshmallow example
import os

from dynamorm import DynaModel, GlobalIndex, ProjectAll

from marshmallow import fields, validate, validates, ValidationError

class Thing(DynaModel):
    class Table:
        name = 'things'
        hash_key = 'id'
        read = 5
        write = 1

    class ByColor(GlobalIndex):
        name = 'by-color'
        hash_key = 'color'
        read = 5
        write = 1
        projection = ProjectAll()

    class Schema:
        id = fields.String(required=True)
        name = fields.String()
        color = fields.String(validate=validate.OneOf(('purple', 'red', 'yellow')))
        compound = fields.Dict(required=True)

        @validates('name')
        def validate_name(self, value):
            # this is a very silly example just to illustrate that you can fill out the
            # inner Schema class just like any other Marshmallow class
            if name.lower() == 'evan':
                raise ValidationError("No Evan's allowed")

    def say_hello(self):
        print("Hello.  {name} here.  My ID is {id} and I'm colored {color}".format(
            id=self.id,
            name=self.name,
            color=self.color
        ))

delete()¶: Delete this record in the table.

classmethod get(consistent=False, **kwargs)¶

Get an item from the table

Example:

Thing.get(hash_key="three")

Parameters

consistent (bool) – If set to True the get will be a consistent read
**kwargs – You must supply your hash key, and range key if used

classmethod get_batch(keys, consistent=False, attrs=None)¶

Generator to get more than one item from the table.

Parameters

keys – One or more dicts containing the hash key, and range key if used
consistent (bool) – If set to True then get_batch will be a consistent read
attrs (str) – The projection expression of which attrs to fetch, if None all attrs will be fetched

classmethod new_from_raw(raw, partial=False)¶

Return a new instance of this model from a raw (dict) of data that is loaded by our Schema

Parameters: raw (dict) – The attributes to use when creating the instance

classmethod put(item, **kwargs)¶

Put a single item into the table for this model

The attributes on the item go through validation, so this may raise ValidationError.

Parameters

item (dict) – The item to put into the table
**kwargs – All other kwargs are passed through to the put method on the table

classmethod put_batch(*items, **batch_kwargs)¶

Put one or more items into the table

Parameters

*items – The items to put into the table
**kwargs – All other kwargs are passed through to the put_batch method on the table

Example:

Thing.put_batch(
    {"hash_key": "one"},
    {"hash_key": "two"},
    {"hash_key": "three"},
)

classmethod put_unique(item, **kwargs)¶

Put a single item into the table for this model, with a unique attribute constraint on the hash key

Parameters

item (dict) – The item to put into the table
**kwargs – All other kwargs are passed through to the put_unique method on the table

classmethod query(*args, **kwargs)¶

Execute a query on our table based on our keys

You supply the key(s) to query based on as keyword arguments:

Thing.query(foo="Mr. Foo")

By default the eq condition is used. If you wish to use any of the other valid conditions for keys use a double underscore syntax following the key name. For example:

Thing.query(foo__begins_with="Mr.")

Parameters

query_kwargs (dict) – Extra parameters that should be passed through to the Table query function
**kwargs – The key(s) and value(s) to query based on

save(partial=False, unique=False, return_all=False, **kwargs)¶

Save this instance to the table

Parameters

partial (bool) – When False the whole document will be .put or .put_unique to the table. When True only values that have changed since the document was loaded will sent to the table via an .update.
unique (bool) – Only relevant if partial=False, ignored otherwise. When False, the document will be .put to the table. When True, the document will be .put_unique.
return_all (bool) – Only used for partial saves. Passed through to .update.
**kwargs – When partial is False these are passed through to the put method on the table. When partial is True these become the kwargs for update_item. See .put & .update for more details.

The attributes on the item go through validation, so this may raise ValidationError.

TODO - Support unique, partial saves.

classmethod scan(*args, **kwargs)¶

Execute a scan on our table

You supply the attr(s) to query based on as keyword arguments:

Thing.scan(age=10)

By default the eq condition is used. If you wish to use any of the other valid conditions for attrs use a double underscore syntax following the key name. For example:

<>: Thing.scan(foo__ne='bar')
<: Thing.scan(count__lt=10)
<=: Thing.scan(count__lte=10)
>: Thing.scan(count__gt=10)
>=: Thing.scan(count__gte=10)
BETWEEN: Thing.scan(count__between=[10, 20])
IN: Thing.scan(count__in=[11, 12, 13])
attribute_exists: Thing.scan(foo__exists=True)
attribute_not_exists: Thing.scan(foo__not_exists=True)
attribute_type: Thing.scan(foo__type='S')
begins_with: Thing.scan(foo__begins_with='f')
contains: Thing.scan(foo__contains='oo')

Accessing nested attributes also uses the double underscore syntax:

Thing.scan(address__state="CA")
Thing.scan(address__state__begins_with="C")

Multiple attrs are combined with the AND (&) operator:

Thing.scan(address__state="CA", address__zip__begins_with="9")

If you want to combine them with the OR (|) operator, or negate them (~), then you can use the Q function and pass them as arguments into scan where each argument is combined with AND:

from dynamorm import Q

Thing.scan(Q(address__state="CA") | Q(address__state="NY"), ~Q(address__zip__contains="5"))

The above would scan for all things with an address.state of (CA OR NY) AND address.zip does not contain 5.

This returns a generator, which will continue to yield items until all matching the scan are produced, abstracting away pagination. More information on scan pagination: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.Pagination

Parameters

scan_kwargs (dict) – Extra parameters that should be passed through to the Table scan function
*args – An optional list of Q objects that can be combined with or superseded the **kwargs values
**kwargs – The key(s) and value(s) to filter based on

update(conditions=None, update_item_kwargs=None, return_all=False, **kwargs)¶

Update this instance in the table

New values are set via kwargs to this function:

thing.update(foo='bar')

This would set the foo attribute of the thing object to 'bar'. You cannot change the Hash or Range key via an update operation – this is a property of DynamoDB.

You can supply a dictionary of conditions that influence the update. In their simpliest form Conditions are supplied as a direct match (eq):

thing.update(foo='bar', conditions=dict(foo='foo'))

This update would only succeed if foo was set to ‘foo’ at the time of the update. If you wish to use any of the other valid conditions for attrs use a double underscore syntax following the key name. You can also access nested attributes using the double underscore syntac. See the scan method for examples of both.

You can also pass Q objects to conditions as either a complete expression, or a list of expressions that will be AND’d together:

thing.update(foo='bar', conditions=Q(foo='foo'))

thing.update(foo='bar', conditions=Q(foo='foo') | Q(bar='bar'))

# the following two statements are equivalent
thing.update(foo='bar', conditions=Q(foo='foo') & ~Q(bar='bar'))
thing.update(foo='bar', conditions=[Q(foo='foo'), ~Q(bar='bar')])

If your update conditions do not match then a dynamorm.exceptions.ConditionFailed exception will be raised.

As long as the update succeeds the attrs on this instance will be updated to match their new values. If you set return_all to true then we will update all of the attributes on the object with the current values in Dyanmo, rather than just those you updated.

classmethod update_item(conditions=None, update_item_kwargs=None, **kwargs)¶

Update a item in the table

Params conditions: A dict of key/val pairs that should be applied as a condition to the update
Params update_item_kwargs: A dict of other kwargs that are passed through to update_item
Params **kwargs: Includes your hash/range key/val to match on as well as any keys to update

validate()¶

Validate this instance

We do this as a “native”/load/deserialization since Marshmallow ONLY raises validation errors for required/allow_none/validate(s) during deserialization. See the note at: https://marshmallow.readthedocs.io/en/latest/quickstart.html#validation

class dynamorm.model.DynaModelMeta(name, parents, attrs)¶

DynaModelMeta is a metaclass for the DynaModel class that transforms our Table and Schema classes

Since we can inspect the data we need to build the full data structures needed for working with tables and indexes users can define for more concise and readable table definitions that we transform into the final. To allow for a more concise definition of DynaModels we do not require that users define their inner Schema class as extending from the Schema. Instead, when the class is being defined we take the inner Schema and transform it into a new class named <Name>Schema, extending from Schema. For example, on a model named Foo the resulting Foo.Schema object would be an instance of a class named FooSchema, rather than a class named Schema

`dynamorm.table`¶

The inner Table class on DynaModel definitions becomes an instance of our dynamorm.table.DynamoTable3 class.

The attributes you define on your inner Table class map to underlying boto data structures. This mapping is expressed through the following data model:

Attribute	Required	Type	Description
name	True	str	The name of the table, as stored in Dynamo.
hash_key	True	str	The name of the field to use as the hash key. It must exist in the schema.
range_key	False	str	The name of the field to use as the range_key, if one is used. It must exist in the schema.
read	True	int	The provisioned read throughput.
write	True	int	The provisioned write throughput.
stream	False	str	The stream view type, either None or one of: ‘NEW_IMAGE’\|’OLD_IMAGE’\|’NEW_AND_OLD_IMAGES’\|’KEYS_ONLY’

Indexes¶

Like the Table definition, Indexes are also inner classes on DynaModel definitions, and they require the same data model with one extra field.

Attribute	Required	Type	Description
projection	True	object	An instance of of `dynamorm.model.ProjectAll`, `dynamorm.model.ProjectKeys`, or `dynamorm.model.ProjectInclude`

class dynamorm.table.DynamoCommon3¶

Common properties & functions of Boto3 DynamORM objects – i.e. Tables & Indexes

property key_schema¶: Return an appropriate KeySchema, based on our key attributes and the schema object

property provisioned_throughput¶: Return an appropriate ProvisionedThroughput, based on our attributes

class dynamorm.table.DynamoGlobalIndex3(table, schema)¶

class dynamorm.table.DynamoIndex3(table, schema)¶

class dynamorm.table.DynamoLocalIndex3(table, schema)¶

class dynamorm.table.DynamoTable3(schema, indexes=None)¶

Represents a Table object in the Boto3 DynamoDB API

This is built in such a way that in the future, when Amazon releases future boto versions, a new DynamoTable class can be authored that implements the same methods but maps through to the new semantics.

property all_attribute_fields¶: Returns a list with the names of all the attribute fields (hash or range key on the table or indexes)

property attribute_definitions¶: Return an appropriate AttributeDefinitions, based on our key attributes and the schema object

create(wait=True)¶: DEPRECATED – shim

create_table(wait=True)¶

Create a new table based on our attributes

Parameters: wait (bool) – If set to True, the default, this call will block until the table is created

delete(wait=True)¶

Delete this existing table

Parameters: wait (bool) – If set to True, the default, this call will block until the table is deleted

property exists¶: Return True or False based on the existance of this tables name in our resource

classmethod get_resource(**kwargs)¶

Return the boto3 resource

If you provide kwargs here and the class doesn’t have any resource_kwargs defined then the ones passed will permanently override the resource_kwargs on the class.

This is useful for bootstrapping test resources against a Dynamo local instance as a call to DynamoTable3.get_resource will end up replacing the resource_kwargs on all classes that do not define their own.

classmethod get_table(name)¶

Return the boto3 Table object for this model, create it if it doesn’t exist

The Table is stored on the class for each model, so it is shared between all instances of a given model.

get_update_expr_for_key(id_, parts)¶

Given a key and a unique id, return all the information required for the update expression. This includes the actual field operations, a dictionary of generated field names, and the generated field value.

To account for nested keys, the generated field expression placeholders are of the form:

#uk_0_0 = :uv_0
#uk_0_0.#uk_0_1 = :uv_0
#uk_0_0.#uk_0_1.#uk_0_2 = :uv_0
...

Note that if the value of a part - e.g #uk_0_1 - itself has a period ., that is interpreted literally and not as nest in the document path. That is:

#uk_0_0.#uk_0_1 = :uv_0
{
    "#uk_0_0": "foo",
    "#uk_0_1": "bar.baz"
}
{
    ":uv_0": 42
}

…will result in the value:

"foo": {
    "bar.baz": 42
}

Parameters

id – Unique id for this key
parts – List of parts that make up this key

Return type

tuple[str, dict, str]

index_attribute_fields(index_name=None)¶: Return the attribute fields for a given index, or all indexes if omitted

put(item, **kwargs)¶

Put a singular item into the table

Parameters

item (dict) – The data to put into the table
**kwargs – All other keyword arguments are passed through to the DynamoDB Table put_item function.

property stream_specification¶: Return an appropriate StreamSpecification, based on the stream attribute

property table¶: Return the boto3 table

property table_attribute_fields¶: Returns a list with the names of the table attribute fields (hash or range key)

update_table()¶

Updates an existing table

Per the AWS documentation:

You can only perform one of the following operations at once:

Modify the provisioned throughput settings of the table.

Enable or disable Streams on the table.

Remove a global secondary index from the table.

Create a new global secondary index on the table.

Thus, this will recursively call itself to perform each of these operations in turn, waiting for the table to return to ‘ACTIVE’ status before performing the next.

This returns the number of update operations performed.

dynamorm.table.Q(**mapping)¶

A Q object represents an AND’d together query using boto3’s Attr object, based on a set of keyword arguments that support the full access to the operations (eq, ne, between, etc) as well as nested attributes.

It can be used input to both scan operations as well as update conditions.

class dynamorm.table.QueryIterator(model, *args, **kwargs)¶

reverse()¶: Return results from the query in reverse

class dynamorm.table.ReadIterator(model, *args, **kwargs)¶

ReadIterator provides an iterator object that wraps a model and a method (either scan or query).

Since it is an object we can attach attributes and functions to it that are useful to the caller.

# Scan through a model, one at a time.  Don't do this!
results = MyModel.scan().limit(1)
for model in results:
    print model.id

# The next time you call scan (or query) pass the .last attribute of your previous results
# in as the last argument
results = MyModel.scan().start(results.last).limit(1)
for model in results:
    print model.id

# ...

Parameters

model – The Model class to wrap
*args – Q objects, passed through to scan or query
**kwargs – filters, passed through to scan or query

again()¶

Call this to reset the iterator so that you can iterate over it again.

If the previous invocation has a LastEvaluatedKey then this will resume from the next item. Otherwise it will re-do the previous invocation.

consistent()¶: Make this read a consistent one

count()¶

Return the count matching the current read

This triggers a new request to the table when it is invoked.

limit(limit)¶: Set the limit value

partial(partial)¶

Set the partial value for this iterator, which is used when creating new items from the response.

This is used by indexes

recursive()¶: Set the recursive value to True for this iterator

specific_attributes(attrs)¶

Return only specific attributes in the documents through a ProjectionExpression

This is a list of attribute names. See the documentation for more info: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ProjectionExpressions.html

start(last)¶: Set the last value

class dynamorm.table.ScanIterator(model, *args, **kwargs)¶

dynamorm.table.remove_nones(in_dict)¶: Recursively remove keys with a value of None from the in_dict collection

`dynamorm.relationships`¶

Relationships leverage the native tables & indexes in DynamoDB to allow more concise definition and access of related objects in your Python code.

You define relationships along side your Schema and Indexes on your model, and must provide the query used to map the related models together. You can also supply a “back reference” query to have the other side of the relationship also have a relationship back to the defining model.

DynamORM provides the following relationship types:

dynamorm.relationships.OneToOne - Useful when you have a large number of attributes to store and you want to break them up over multiple tables for performance in querying.
dynamorm.relationships.OneToMany / dynamorm.relationships.ManyToOne - Useful when you have an instance of one model that has a collection of related instances of another model. You use OneToMany or ManyToOne depending on which side of the relationship you are defining the attribute on. You’ll to use both interchangeably based on how your models are laid out since you need to pass a reference to the other model into the relationship.

Here’s an example of how you could model the Forum Application from the DynamoDB Examples:

class Reply(DynaModel):
    class Table:
        name = 'replies'
        hash_key = 'forum_thread'
        range_key = 'created'
        read = 1
        write = 1

    class ByUser(GlobalIndex):
        name = 'replies-by-user'
        hash_key = 'user_name'
        range_key = 'message'
        projection = ProjectKeys()
        read = 1
        write = 1

    class Schema:
        forum_thread = String(required=True)
        created = String(required=True)
        user_name = String(required=True)
        message = String()

class User(DynaModel):
    class Table:
        name = 'users'
        hash_key = 'name'
        read = 1
        write = 1

    class Schema:
        name = String(required=True)

    replies = OneToMany(
        Reply,
        index='ByUser',
        query=lambda user: dict(user_name=user.name),
        back_query=lambda reply: dict(name=reply.user_name)
    )

class Thread(DynaModel):
    class Table:
        name = 'threads'
        hash_key = 'forum_name'
        range_key = 'subject'
        read = 1
        write = 1

    class ByUser(GlobalIndex):
        name = 'threads-by-user'
        hash_key = 'user_name'
        range_key = 'subject'
        projection = ProjectKeys()
        read = 1
        write = 1

    class Schema:
        forum_name = String(required=True)
        user_name = String(required=True)
        subject = String(required=True)

    user = ManyToOne(
        User,
        query=lambda thread: dict(name=thread.user_name),
        back_index='ByUser',
        back_query=lambda user: dict(user_name=user.name)
    )
    replies = OneToMany(
        Reply,
        query=lambda thread: dict(forum_thread='{0}\n{1}'.format(thread.forum_name, thread.subject)),
        back_query=lambda reply: dict(
            forum_name=reply.forum_thread.split('\n')[0],
            subject=reply.forum_thread.split('\n')[1]
        )
    )

class Forum(DynaModel):
    class Table:
        name = 'forums'
        hash_key = 'name'
        read = 1
        write = 1

    class Schema:
        name = String(required=True)

    threads = OneToMany(
        Thread,
        query=lambda forum: dict(forum_name=forum.name),
        back_query=lambda thread: dict(name=thread.forum_name)
    )

class dynamorm.relationships.DefaultBackReference(relationship)¶

When given a relationship the string representation of this will be a “python” string name of the model the relationship exists on.

For example, if there’s a relationship defined on a model named OrderItem this would render order_item.

class dynamorm.relationships.ManyToOne(other, query, index=None, back_query=None, back_index=None, back_reference=<class 'dynamorm.relationships.DefaultBackReference'>, auto_create=True)¶

A Many To One relationship is defined on the “child” model, where many child models have one parent model.

BackReferenceClass¶: alias of OneToMany

class dynamorm.relationships.OneToMany(other, query, index=None, back_query=None, back_index=None, back_reference=<class 'dynamorm.relationships.DefaultBackReference'>)¶

A One to Many relationship is defined on the “parent” model, where each instance has many related “child” instances of another model.

BackReferenceClass¶: alias of OneToOne

class dynamorm.relationships.OneToOne(other, query, index=None, back_query=None, back_index=None, back_reference=<class 'dynamorm.relationships.DefaultBackReference'>, auto_create=True)¶

A One-to-One relationship is where two models (tables) have items that have a relation to exactly one instance in the other model.

It is a useful pattern when you wish to split up large tables with many attributes where your “main” table is queried frequently and having all of the attributes included in the query results would increase your required throughput. By splitting the data into two tables you can have lower throughput on the “secondary” table as the items will be lazily fetched only as they are accessed.

set_this_model(model)¶: Called from the metaclass once the model the relationship is being placed on has been initialized

`dynamorm.signals`¶

Signals provide a way for applications to loosely couple themselves and respond to different life cycle events.

The blinker library provides the low-level signal implementation.

To use the signals you connect a receiver function to the signals you’re interested in:

from dynamorm.signals import post_save

def post_save_receiver(sender, instance, partial, put_kwargs):
    log.info("Received post_save signal from model %s for instance %s", sender, instance)

post_save.connect(post_save_receiver)

See the blinker documentation for more details.

dynamorm.signals.model_prepared¶

Sent whenever a model class has been prepared by the metaclass.

Param: sender: The model class that is now prepared for use.

dynamorm.signals.pre_init¶

Sent during model instantiation, before processing the raw data.

Param: sender: The model class.
Param: instance: The model instance.
Param: bool partial: True if this is a partial instantiation, not all data may be present.
Param: dict raw: The raw data to be processed by the model schema.

dynamorm.signals.post_init¶

Sent once model instantiation is complete and all raw data has been processed.

Param: sender: The model class.
Param: instance: The model instance.
Param: bool partial: True if this is a partial instantiation, not all data may be present.
Param: dict raw: The raw data to be processed by the model schema.

dynamorm.signals.pre_save¶

Sent before saving (via put) model instances.

Param: sender: The model class.
Param: instance: The model instance.
Param: dict put_kwargs: A dict of the kwargs being sent to the table put method.

dynamorm.signals.post_save¶

Sent after saving (via put) model instances.

Param: sender: The model class.
Param: instance: The model instance.
Param: dict put_kwargs: A dict of the kwargs being sent to the table put method.

dynamorm.signals.pre_update¶

Sent before saving (via update) model instances.

Param: sender: The model class.
Param: instance: The model instance.
Param: dict conditions: The conditions for the update to succeed.
Param: dict update_item_kwargs: A dict of the kwargs being sent to the table put method.
Param: dict updates: The fields to update.

dynamorm.signals.post_update¶

Sent after saving (via update) model instances.

Param: sender: The model class.
Param: instance: The model instance.
Param: dict conditions: The conditions for the update to succeed.
Param: dict update_item_kwargs: A dict of the kwargs being sent to the table put method.
Param: dict updates: The fields to update.

dynamorm.signals.pre_delete¶

Sent before deleting model instances.

Param: sender: The model class.
Param: instance: The model instance.

dynamorm.signals.post_delete¶

Sent after deleting model instances.

Param: sender: The model class.
Param: instance: The deleted model instance.

`dynamorm.exceptions`¶

exception dynamorm.exceptions.ConditionFailed¶: A condition check failed

exception dynamorm.exceptions.DynaModelException¶: Base exception for DynaModel problems

exception dynamorm.exceptions.DynamoException¶: Base exception for all DynamORM raised exceptions

exception dynamorm.exceptions.DynamoTableException¶: Base exception class for all DynamoTable errors

exception dynamorm.exceptions.HashKeyExists¶: A operating requesting a unique hash key failed

exception dynamorm.exceptions.InvalidKey¶: A parameter is not a valid key

exception dynamorm.exceptions.InvalidSchemaField¶: A field provided does not exist in the schema

exception dynamorm.exceptions.MissingTableAttribute¶: A required attribute is missing

exception dynamorm.exceptions.TableNotActive¶: The table is not ACTIVE, and you do not want to wait

exception dynamorm.exceptions.ValidationError(raw, schema_name, errors, *args, **kwargs)¶: Schema validation failed

`dynamorm.local`¶

class dynamorm.local.DynamoLocal(dynamo_dir, port=None)¶: Spins up a local dynamo instance. This should ONLY be used for testing!! This instance will register the cleanup method shutdown with the atexit module.

dynamorm.local.get_random_port()¶: Find a random port that appears to be available

DynamORM API¶

`dynamorm`¶

`dynamorm.model`¶

`dynamorm.table`¶

Indexes¶

`dynamorm.relationships`¶

`dynamorm.signals`¶

`dynamorm.exceptions`¶

`dynamorm.local`¶

DynamORM

Navigation

Related Topics

DynamORM API¶

dynamorm¶

dynamorm.model¶

dynamorm.table¶

Indexes¶

dynamorm.relationships¶

dynamorm.signals¶

dynamorm.exceptions¶

dynamorm.local¶

`dynamorm`¶

`dynamorm.model`¶

`dynamorm.table`¶

`dynamorm.relationships`¶

`dynamorm.signals`¶

`dynamorm.exceptions`¶

`dynamorm.local`¶