Skip to content

Updated DynamoDB enhanced client design document. #1256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 24, 2019
Merged

Updated DynamoDB enhanced client design document. #1256

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
55e9d4e
Updated DynamoDB enhanced client design document.
millems May 13, 2019
7f7b1a9
Merge branch 'master' into millem/dynamodb-enhanced-docs
millems May 14, 2019
b5d1acc
Merge branch 'master' into millem/dynamodb-enhanced-docs
millems May 14, 2019
cea6114
Merge branch 'master' into millem/dynamodb-enhanced-docs
millems May 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
374 changes: 331 additions & 43 deletions docs/design/services/dynamodb/high-level-library/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,347 @@
**Design:** New Feature, **Status:**
[In Development](../../../README.md)

# SDK Tenets (unless you know better ones)
## Tenets (unless you know better ones)

1. Meeting customers in their problem space allows them to deliver value
quickly.
2. Meeting customer expectations drives usability.
3. Discoverability drives usage.
4. Providing a Java-focused experience for DynamoDB reduces the coding
effort required to integrate with DynamoDB.
5. Reusing the same nouns and verbs as the generated DynamoDB client
meets customer expectations.
6. Optimizing for cold-start performance allows customers the
convenience of using object mapping in a Lambda environment.

# Project Tenets (unless you know better ones)
## Problem

1. Providing a Java-focused experience for DynamoDB reduces the coding
effort required to integrate with DynamoDB.
2. Reusing the same nouns and verbs as the generated DynamoDB client
meets customer expectations.
3. Optimizing for cold-start performance allows customers the
convenience of using object mapping in a Lambda environment.
Customers on the AWS SDK for Java 2.x currently use the `DynamoDbClient`
to communicate with DynamoDB. This client is generated from the model
provided by the DynamoDB team.

Because this client is generated, it does not provide an idiomatic Java
experience. For example: (1) the client represents numbers as `String`
instead of the more idiomatic `Number`, (2) customers must manually
convert common Java data types like `Instant` into types supported by
DynamoDB, (3) customers that represent their DynamoDB objects using Java
objects must manually convert these objects into the item representation
supported by DynamoDB.

## Existing Solutions

This problem is not currently addressed directly in the AWS SDK for Java
2.x by any known third-party tool. In 1.11.x, several solutions exist,
including AWS's own Document and Mapper Clients.

## Proposed Solution

The AWS SDK for Java will add a new "enhanced DynamoDB client" that
provides an alternative to the data-access portion of the generated
DynamoDB APIs. Control-plane operations like "create table" will not be
supported at launch, but may be added at a later time.

This enhanced client will make DynamoDB easier to use for Java customers
by:
1. Supporting conversions between Java objects and DynamoDB items
2. Supporting conversions between Java built-in types (eg. `Instant`)
and DynamoDB attribute value types
3. Directly supporting every data-plane operation of DynamoDB
4. Using the same verbs and nouns of DynamoDB

## Implementation Overview

**New Clients**

Two new client classes will be added:
`DynamoDbEnhancedClient` and `DynamoDbEnhancedAsyncClient`. These
classes act as a wrapper around the generated `DynamoDbClient` and
`DynamoDbAsyncClient` classes, to provide additional functionality on
top of that which can be provided by the generated clients.

```java
DynamoDbEnhancedClient enhancedClient =
DynamoDbEnhancedClient.builder()
.dynamoDbClient(DynamoDbClient.create())
.build();
```

**Table Abstraction**

`DynamoDbEnhancedClient` provides access to `Table` and `MappedTable`,
and `DynamoDbEnhancedAsyncClient`provides access to `AsyncTable`, and
`AsyncMappedTable` abstractions.

The operations on these "tables" match the data-plane operations in the
low-level DynamoDB client. For example, because `DynamoDbClient.putItem`
exists, `Table.putItem` will also exist.

`Table` and `AsyncTable` work with "items", described below.
`MappedTable` and `AsyncMappedTable` work with "objects", described
below. `Table` and `MappedTable` returning results synchronously, and
`AsyncTable` and `AsyncMappedTable` returning results asynchronously.

```java
Table booksTable = enhancedClient.table("books");
booksTable.putItem(...);

MappedTable mappedBooksTable = enhancedClient.mappedTable("books");
mappedBooksTable.putItem(...);
```

**Item Abstraction**

The operations on `Table` and `AsyncTable` work on `Item`s. An `Item` is
a user-friendly representation of the generated `Map<String,
AttributeValue>`. `Item`s support automatic type conversion between Java
built-in types and DynamoDB-specific `AttributeValue` types.

```java
booksTable.putItem(Item.builder()
.putAttribute("isbn", "0-330-25864-8")
.putAttribute("title", "The Hitchhiker's Guide to the Galaxy")
.putAttribute("creationDate", Instant.now())
.build());
```

The `Table` and `AsyncTable` abstractions can be seen as a replacement
for the 1.11.x DynamoDB Document client.

**Object Abstraction**

The operations on `MappedTable` and `AsyncMappedTable` work on Java
objects (at launch, Java beans). These objects are automatically
converted by the enhanced client to the generated `Map<String,
AttributeValue>`. It's likely that the `MappedTable` and
`AsyncMappedTable` will use the `Table` and `AsyncTable` as an
implementation detail.

```java
Book book = new Book();
book.setIsbn("0-330-25864-8");
book.setTitle("The Hitchhiker's Guide to the Galaxy");
book.setCreationDate(Instant.now());
mappedBooksTable.putItem(book);
```

The `MappedTable` and `AsyncMappedTable` abstractions can be seen as a
replacement for the 1.11.x DynamoDB Mapper client.

**Type Conversion**

The core feature of the mapper is the ability to convert common Java
structures (e.g. Java beans) and types (e.g. `Instant`, `Number`) into
DynamoDB attribute values.

These conversions are performed based on the types specified by the
customer. For example, the SDK will automatically convert any `Number`
types specified by the customer (as an Item attribute) into a DynamoDB
number.

The customer has the ability to configure the type converters used at
the `Item` or `DynamoDbEnhanced[Async]Client`-level. This allows the
customer to add support for unsupported types, change the DynamoDB type
associated with a Java type (e.g. storing an `Instant` as a DynamoDB
string instead of a number), or to add support for custom POJO
conversion logic (i.e. other than Java beans). This also allows the
customer to provide a hard-coded converter for a specific object type
that performs more efficiently than the built-in reflection-based object
converter.

## Features

**Launch Features**

# Project Introduction
These features are intended for inclusion at launch of the library.

The enhanced DynamoDB client replaces the generated DynamoDB client with
one that is easier for a Java customer to use. It does this by
supporting conversions between Java objects and DynamoDB items, as well
as converting between Java built-in types (eg. java.time.Instant) and
DynamoDB attribute value types.
1. Support for all existing data plane operations: get, put, query,
update, scan, delete, batch get, batch put, transaction get, and
transaction put.
2. Support for `[Async]Table` and `[Async]MappedTable`, as described
above.
3. Support for bean-based representations in `[Async]MappedTable`.
4. Type converters for all Java built-in types that are currently
supported by [Joda Convert](https://www.joda.org/joda-convert/).

The enhanced DynamoDB client intentionally does not attempt to simplify
specific data access patterns, like relational or time-series data. It
is within the scope of future projects to provide data access
pattern-specific abstractions on top of the enhanced DynamoDB client
and/or other AWS services.
| API | Feature | Development | Usability Study |
| --- | --- | --- | --- |
| Item | Get | Done | |
| | Put | Done | |
| | Query | | |
| | Update | | |
| | Scan | | |
| | Delete | | |
| | Batch Get | | |
| | Batch Put | | |
| | Transaction Get | | |
| | Transaction Put | | |
| Object | Get | | |
| | Put | | |
| | Query | | |
| | Update | | |
| | Scan | | |
| | Delete | | |
| | Batch Get | | |
| | Batch Put | | |
| | Transaction Get | | |
| | Transaction Put | | |
| All | Type Support | In Progress | |

# Links
**Post-Launch Features**

1. Support for inheritance in `[Async]MappedTable`.
2. Support for immutable objects in `[Async]MappedTable`.
3. Support for projection statements in `[Async]Table` and
`[Async]MappedTable`.
4. Support for DynamoDB-provided API metrics (e.g. consumed capacity).
5. A `software.amazon.aws:dynamodb-all` module that automatically
includes all AWS DynamoDB artifacts, to enhance client
discoverability.

**Missing Features**

These features are not intended for inclusion at launch of the library
(but may be added at a future time).

1. Support for control-plane operations, like create or delete table.
*Justification for exclusion:* For testing purposes, this can be done
through the AWS console or low-level SDK. For production purposes,
this should be done through the AWS CDK or cloud formation.
2. Versioning and UUID annotations. *Justification for exclusion:* This
is a higher-level concern than the "type converter" goal that the
enhanced client is attempting to deliver on. This is a piece of
functionality that will be built on-top of the enhanced client, not
in it.

**Requested Features**

* [Immutable classes](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-315049138)
* [Getter/setter-less fields](https://github.com/aws/aws-sdk-java/issues/547)
* [Replace `PaginatedList` with `Stream`](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-318051305)
* [Allow 'setters' and 'getters' to support different types](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-318792534)
* [Have 'scan' respect the table's read throughput](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-329007523)
* [Allow creating a table with an LSI that projects all attributes](https://github.com/aws/aws-sdk-java/issues/214#issue-31304615)
* [Projection expressions in 'load' and 'batchLoad'](https://github.com/aws/aws-sdk-java/issues/527)
* [New condition expressions](https://github.com/aws/aws-sdk-java/issues/534)
* [Accessing un-modeled/dynamic attributes in a POJO](https://github.com/aws/aws-sdk-java/issues/674)
* [Inheritance](https://github.com/aws/aws-sdk-java/issues/832)
* [Service-side metrics](https://github.com/aws/aws-sdk-java/issues/953)
([1](https://github.com/aws/aws-sdk-java/issues/1170),
[2](https://github.com/aws/aws-sdk-java-v2/issues/703),
[3](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-417656448))
* [Merging DynamoDB mapper configurations](https://github.com/aws/aws-sdk-java/issues/1201)
* [Cache merged DynamoDB mapper configurations](https://github.com/aws/aws-sdk-java/issues/1235)
* [Create one single type converter interface](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-330616648)
* [Support `@DynamoDBGeneratedUuid` in objects nested within lists](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332958299)
* [Allow annotating fields in addition to methods](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332968651)
* [Non-string keys in maps](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332974427)
* [Multiple conditions on the same attribute, for save/delete](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-342586344)
* [Persisting public getters from package-private classes](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-343006566)
* [Return modified attributes when doing a save](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-417656448)
* [More direct exposure of scan or filter expressions](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-430993224)
* [Transactions support](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-443308198)
* [Creating an Item from JSON (and vice-versa)](https://github.com/aws/aws-sdk-java-v2/issues/1240)
* Straight-forward support for multiple classes in a single table (as
per
[here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html))
(from email)
* Support for `Optional` (from email)
* Support for `Publisher` for async paginated responses (from email)
* Create a table with partial projections (from email)
* Better integration with DynamoDB streams (from email)
* Configuring table auto-scaling when a table is created (from email)
* Request-level credentials (from email)
* Wrappers for transactional isolation (from email)
* Dynamic attributes - ones with different types depending on the value
of other attributes, or attributes with names that are generated at
runtime (from email)
* Structure versioning (from email)

## Appendix A: Alternative Solutions

### Alternative Solution 1: Level 3 Storage Library

A "Level 2" high-level library is a service-specific library built on
top of the "Level 1" generated client. The solution proposed above is a
Level 2 high-level library for DynamoDB.

A "Level 3" high-level library focuses on a specific customer problem
instead of a specific AWS service. For example, customers frequently use
DynamoDB to store time series data. An alternate to the proposed
solution above, would be to build multiple Level 3 libraries, each
focusing on a specific customer problem: a document database library, a
time series database library, etc. These libraries would support
DynamoDB as one of many backing data stores.

Instead of using traditional DynamoDB nouns and verbs (e.g. Item), a
Level 3 library would use words more aligned to the problem domain (e.g.
Document for document databases or Entry for time-series data). They
would also expose operations more constrained to the problem domain they
were trying to solve, instead of trying to expose every piece of
DynamoDB functionality.

This solution would be better for customers that are more familiar with
the problem they are trying to solve and less familiar with DynamoDB.
This solution would be worse for customers that are familiar with
DynamoDB and want to be "closer" to the service.

**Customer Feedback**

The Java SDK team collected customer feedback internally and
[externally](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-468435660),
comparing this alternate solution against the proposed solution.
Customers were presented with the following option comparison:

> Option 1: A DynamoDB-specific client that combines the functionality
> of 1.11.x's Documents APIs and DynamoDB Mapper APIs in a
> straight-forward manner.

> Option 2: A generic document database client that creates an
> abstraction over all document databases, like DynamoDB and MongoDB.
> This would simplify using multiple document databases in the same
> application, and make it easier to migrate between the two.
> Unfortunately as a result, it also wouldn't be a direct DynamoDB
> experience.

We requested that customers review these two options as well as a
[prototype of option 1](prototype/option-1/sync/Prototype.java) and a
[prototype of option 2](prototype/option-2/sync/Prototype.java), to let
us know which they prefer.

The following anecdotes are from this customer feedback:

> If \[Amazon] can make something like https://serverless.com/ or
> https://onnx.ai/ which free customers from vendor lock-in, that would
> be a great Think Big & Customer Obsession idea. If \[Amazon] cannot,
> I feel that somebody who is more vendor-neutral can make a better
> mapper than \[Amazon].

> Have you thought about contributing to projects which already exist,
> like Spring Data? https://github.com/derjust/spring-data-dynamodb

> Both options would work well for us.

> I think \[doing option 1 and then creating a Spring Data plugin] might
> get adoption from a broader audience than option 2. It could be used
> as a stepping stone to move to DynamoDB.

> I believe Option 2 does not make much sense. It would make sense to me
> to go for Option 1 and start a bounty program to implement a module to
> popular data access abstraction libraries such as spring-data
> mentioned above or GORM.

> Maybe you could implement/support JNOSQL spec http://www.jnosql.org/

**Decision**

Based on customer feedback, it was decided to temporarily reject
alternative solution 1, and build the proposed solution. At a later
time, the SDK may build a Level 3 abstraction for DynamoDB or integrate
with existing Java Level 3 abstractions like Spring Data, Hibernate OGM,
and/or JNoSQL. This Level 3 abstraction will possibly leverage the Level
2 solution "under the hood".

## Links

**[Features](features.md)** - The features intended for inclusion during
and after the launch of the enhanced DynamoDB client.
Expand All @@ -55,26 +365,4 @@ to solicit feedback from customers on potential design directions.
for DynamoDB mapper-equivalent functionality in 2.x.
* [DynamoDB Document API Feature Request](https://github.com/aws/aws-sdk-java-v2/issues/36)
\- A github issue for tracking customer feature requests and feedback
for DynamoDB document API-equivalent functionality in 2.x.

# FAQ

**Why not optimize for specific data access patterns, like relational
data?**

Some customers prefer to think about the DynamoDB-specific concepts of
tables, queries, conditions and global secondary indices. These
customers currently have trouble interacting with DynamoDB, because they
have to perform a large amount of conversion between their Java types
and DynamoDB types.

If we were to optimize this specific project for a particular access
pattern, customers that have other access patterns would either be
forced to modify their access pattern into the one we've optimized for,
or they would need to use the generated client and handle their own Java
type conversion.

Instead, this project focuses on making integrating with DynamoDB easier
to do in Java, and leaves the problems of "making storing time series
data on AWS easy" or "making storing documents on AWS easy" to another
project.
for DynamoDB document API-equivalent functionality in 2.x.