Updated DynamoDB enhanced client design document.

millems · millems · commit b948ab9ec955 · 2019-05-24T15:31:38.000-07:00
diff --git a/docs/design/services/dynamodb/high-level-library/README.md b/docs/design/services/dynamodb/high-level-library/README.md
@@ -1,37 +1,347 @@
 **Design:** New Feature, **Status:**
 [In Development](../../../README.md)
 
-# SDK Tenets (unless you know better ones)
+## Tenets (unless you know better ones)
 
 1. Meeting customers in their problem space allows them to deliver value
    quickly.
 2. Meeting customer expectations drives usability.
 3. Discoverability drives usage.
+4. Providing a Java-focused experience for DynamoDB reduces the coding
+   effort required to integrate with DynamoDB. 
+5. Reusing the same nouns and verbs as the generated DynamoDB client
+   meets customer expectations. 
+6. Optimizing for cold-start performance allows customers the
+   convenience of using object mapping in a Lambda environment.
 
-# Project Tenets (unless you know better ones)
+## Problem
 
-1. Providing a Java-focused experience for DynamoDB reduces the coding 
-   effort required to integrate with DynamoDB.
-2. Reusing the same nouns and verbs as the generated DynamoDB client
-   meets customer expectations.
-3. Optimizing for cold-start performance allows customers the
-   convenience of using object mapping in a Lambda environment.
+Customers on the AWS SDK for Java 2.x currently use the `DynamoDbClient`
+to communicate with DynamoDB. This client is generated from the model
+provided by the DynamoDB team.
+
+Because this client is generated, it does not provide an idiomatic Java
+experience. For example: (1) the client represents numbers as `String`
+instead of the more idiomatic `Number`, (2) customers must manually
+convert common Java data types like `Instant` into types supported by
+DynamoDB, (3) customers that represent their DynamoDB objects using Java
+objects must manually convert these objects into the item representation
+supported by DynamoDB.
+
+## Existing Solutions
+
+This problem is not currently addressed directly in the AWS SDK for Java
+2.x by any known third-party tool. In 1.11.x, several solutions exist,
+including AWS's own Document and Mapper Clients.
+
+## Proposed Solution
+
+The AWS SDK for Java will add a new "enhanced DynamoDB client" that
+provides an alternative to the data-access portion of the generated
+DynamoDB APIs. Control-plane operations like "create table" will not be
+supported at launch, but may be added at a later time.
+
+This enhanced client will make DynamoDB easier to use for Java customers
+by:
+1. Supporting conversions between Java objects and DynamoDB items
+2. Supporting conversions between Java built-in types (eg. `Instant`)
+   and DynamoDB attribute value types
+3. Directly supporting every data-plane operation of DynamoDB
+4. Using the same verbs and nouns of DynamoDB
+
+## Implementation Overview
+
+**New Clients**
+
+Two new client classes will be added:
+`DynamoDbEnhancedClient` and `DynamoDbEnhancedAsyncClient`. These
+classes act as a wrapper around the generated `DynamoDbClient` and
+`DynamoDbAsyncClient` classes, to provide additional functionality on
+top of that which can be provided by the generated clients.
+
+```java
+DynamoDbEnhancedClient enhancedClient = 
+    DynamoDbEnhancedClient.builder()
+                          .dynamoDbClient(DynamoDbClient.create())
+                          .build();
+```
+
+**Table Abstraction**
+
+`DynamoDbEnhancedClient` provides access to `Table` and `MappedTable`,
+and `DynamoDbEnhancedAsyncClient`provides access to `AsyncTable`, and
+`AsyncMappedTable` abstractions.
+
+The operations on these "tables" match the data-plane operations in the
+low-level DynamoDB client. For example, because `DynamoDbClient.putItem`
+exists, `Table.putItem` will also exist.
+
+`Table` and `AsyncTable` work with "items", described below.
+`MappedTable` and `AsyncMappedTable` work with "objects", described
+below. `Table` and `MappedTable` returning results synchronously, and
+`AsyncTable` and `AsyncMappedTable` returning results asynchronously.
+
+```java
+Table booksTable = enhancedClient.table("books");
+booksTable.putItem(...);
+
+MappedTable mappedBooksTable = enhancedClient.mappedTable("books");
+mappedBooksTable.putItem(...);
+```
+ 
+**Item Abstraction**
+
+The operations on `Table` and `AsyncTable` work on `Item`s. An `Item` is
+a user-friendly representation of the generated `Map<String,
+AttributeValue>`. `Item`s support automatic type conversion between Java
+built-in types and DynamoDB-specific `AttributeValue` types.
+
+```java
+booksTable.putItem(Item.builder()
+                       .putAttribute("isbn", "0-330-25864-8")
+                       .putAttribute("title", "The Hitchhiker's Guide to the Galaxy")
+                       .putAttribute("creationDate", Instant.now())
+                       .build());
+```
+
+The `Table` and `AsyncTable` abstractions can be seen as a replacement
+for the 1.11.x DynamoDB Document client.
+
+**Object Abstraction**
+
+The operations on `MappedTable` and `AsyncMappedTable` work on Java
+objects (at launch, Java beans). These objects are automatically
+converted by the enhanced client to the generated `Map<String,
+AttributeValue>`. It's likely that the `MappedTable` and
+`AsyncMappedTable` will use the `Table` and `AsyncTable` as an
+implementation detail.
+
+```java
+Book book = new Book();
+book.setIsbn("0-330-25864-8");
+book.setTitle("The Hitchhiker's Guide to the Galaxy");
+book.setCreationDate(Instant.now());
+mappedBooksTable.putItem(book);
+```
+
+The `MappedTable` and `AsyncMappedTable` abstractions can be seen as a
+replacement for the 1.11.x DynamoDB Mapper client.
+
+**Type Conversion**
+
+The core feature of the mapper is the ability to convert common Java
+structures (e.g. Java beans) and types (e.g. `Instant`, `Number`) into
+DynamoDB attribute values. 
+
+These conversions are performed based on the types specified by the
+customer. For example, the SDK will automatically convert any `Number`
+types specified by the customer (as an Item attribute) into a DynamoDB
+number.
+
+The customer has the ability to configure the type converters used at
+the `Item` or `DynamoDbEnhanced[Async]Client`-level. This allows the
+customer to add support for unsupported types, change the DynamoDB type
+associated with a Java type (e.g. storing an `Instant` as a DynamoDB
+string instead of a number), or to add support for custom POJO
+conversion logic (i.e. other than Java beans). This also allows the
+customer to provide a hard-coded converter for a specific object type
+that performs more efficiently than the built-in reflection-based object
+converter.
+
+## Features
+
+**Launch Features**
 
-# Project Introduction
+These features are intended for inclusion at launch of the library.
 
-The enhanced DynamoDB client replaces the generated DynamoDB client with
-one that is easier for a Java customer to use. It does this by
-supporting conversions between Java objects and DynamoDB items, as well
-as converting between Java built-in types (eg. java.time.Instant) and
-DynamoDB attribute value types.
+1. Support for all existing data plane operations: get, put, query,
+   update, scan, delete, batch get, batch put, transaction get, and
+   transaction put.
+2. Support for `[Async]Table` and `[Async]MappedTable`, as described
+   above.
+3. Support for bean-based representations in `[Async]MappedTable`.
+4. Type converters for all Java built-in types that are currently 
+   supported by [Joda Convert](https://www.joda.org/joda-convert/).
 
-The enhanced DynamoDB client intentionally does not attempt to simplify
-specific data access patterns, like relational or time-series data. It
-is within the scope of future projects to provide data access
-pattern-specific abstractions on top of the enhanced DynamoDB client
-and/or other AWS services.
+| API | Feature | Development | Usability Study |
+| --- | --- | --- | --- |
+| Item | Get | Done | |
+| | Put | Done | |
+| | Query | | |
+| | Update | | |
+| | Scan | | |
+| | Delete | | |
+| | Batch Get | | |
+| | Batch Put | | |
+| | Transaction Get | | |
+| | Transaction Put | | |
+| Object | Get | | |
+| | Put | | |
+| | Query | | |
+| | Update | | |
+| | Scan | | |
+| | Delete | | |
+| | Batch Get | | |
+| | Batch Put | | |
+| | Transaction Get | | |
+| | Transaction Put | | |
+| All | Type Support | In Progress | |
 
-# Links
+**Post-Launch Features**
+
+1. Support for inheritance in `[Async]MappedTable`.
+2. Support for immutable objects in `[Async]MappedTable`.
+3. Support for projection statements in `[Async]Table` and
+   `[Async]MappedTable`.
+4. Support for DynamoDB-provided API metrics (e.g. consumed capacity).
+5. A `software.amazon.aws:dynamodb-all` module that automatically
+   includes all AWS DynamoDB artifacts, to enhance client
+   discoverability.
+
+**Missing Features**
+
+These features are not intended for inclusion at launch of the library
+(but may be added at a future time).
+
+1. Support for control-plane operations, like create or delete table.
+   *Justification for exclusion:* For testing purposes, this can be done
+   through the AWS console or low-level SDK. For production purposes,
+   this should be done through the AWS CDK or cloud formation.
+2. Versioning and UUID annotations. *Justification for exclusion:* This
+   is a higher-level concern than the "type converter" goal that the
+   enhanced client is attempting to deliver on. This is a piece of
+   functionality that will be built on-top of the enhanced client, not
+   in it.
+   
+**Requested Features**
+
+* [Immutable classes](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-315049138)
+* [Getter/setter-less fields](https://github.com/aws/aws-sdk-java/issues/547)
+* [Replace `PaginatedList` with `Stream`](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-318051305)
+* [Allow 'setters' and 'getters' to support different types](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-318792534)
+* [Have 'scan' respect the table's read throughput](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-329007523)
+* [Allow creating a table with an LSI that projects all attributes](https://github.com/aws/aws-sdk-java/issues/214#issue-31304615)
+* [Projection expressions in 'load' and 'batchLoad'](https://github.com/aws/aws-sdk-java/issues/527)
+* [New condition expressions](https://github.com/aws/aws-sdk-java/issues/534)
+* [Accessing un-modeled/dynamic attributes in a POJO](https://github.com/aws/aws-sdk-java/issues/674)
+* [Inheritance](https://github.com/aws/aws-sdk-java/issues/832)
+* [Service-side metrics](https://github.com/aws/aws-sdk-java/issues/953)
+  ([1](https://github.com/aws/aws-sdk-java/issues/1170),
+  [2](https://github.com/aws/aws-sdk-java-v2/issues/703),
+  [3](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-417656448))
+* [Merging DynamoDB mapper configurations](https://github.com/aws/aws-sdk-java/issues/1201)
+* [Cache merged DynamoDB mapper configurations](https://github.com/aws/aws-sdk-java/issues/1235)
+* [Create one single type converter interface](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-330616648)
+* [Support `@DynamoDBGeneratedUuid` in objects nested within lists](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332958299)
+* [Allow annotating fields in addition to methods](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332968651)
+* [Non-string keys in maps](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-332974427)
+* [Multiple conditions on the same attribute, for save/delete](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-342586344)
+* [Persisting public getters from package-private classes](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-343006566)
+* [Return modified attributes when doing a save](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-417656448)
+* [More direct exposure of scan or filter expressions](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-430993224)
+* [Transactions support](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-443308198)
+* [Creating an Item from JSON (and vice-versa)](https://github.com/aws/aws-sdk-java-v2/issues/1240)
+* Straight-forward support for multiple classes in a single table (as
+  per
+  [here](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-general-nosql-design.html))
+  (from email)
+* Support for `Optional` (from email)
+* Support for `Publisher` for async paginated responses (from email)
+* Create a table with partial projections (from email)
+* Better integration with DynamoDB streams (from email)
+* Configuring table auto-scaling when a table is created (from email)
+* Request-level credentials (from email)
+* Wrappers for transactional isolation (from email)
+* Dynamic attributes - ones with different types depending on the value
+  of other attributes, or attributes with names that are generated at
+  runtime (from email)
+* Structure versioning (from email)
+
+## Appendix A: Alternative Solutions
+
+### Alternative Solution 1: Level 3 Storage Library
+
+A "Level 2" high-level library is a service-specific library built on
+top of the "Level 1" generated client. The solution proposed above is a
+Level 2 high-level library for DynamoDB.
+
+A "Level 3" high-level library focuses on a specific customer problem
+instead of a specific AWS service. For example, customers frequently use
+DynamoDB to store time series data. An alternate to the proposed
+solution above, would be to build multiple Level 3 libraries, each
+focusing on a specific customer problem: a document database library, a
+time series database library, etc. These libraries would support
+DynamoDB as one of many backing data stores.
+
+Instead of using traditional DynamoDB nouns and verbs (e.g. Item), a
+Level 3 library would use words more aligned to the problem domain (e.g.
+Document for document databases or Entry for time-series data). They
+would also expose operations more constrained to the problem domain they
+were trying to solve, instead of trying to expose every piece of
+DynamoDB functionality.
+
+This solution would be better for customers that are more familiar with
+the problem they are trying to solve and less familiar with DynamoDB.
+This solution would be worse for customers that are familiar with
+DynamoDB and want to be "closer" to the service.
+
+**Customer Feedback**
+
+The Java SDK team collected customer feedback internally and
+[externally](https://github.com/aws/aws-sdk-java-v2/issues/35#issuecomment-468435660),
+comparing this alternate solution against the proposed solution.
+Customers were presented with the following option comparison:
+
+> Option 1: A DynamoDB-specific client that combines the functionality
+> of 1.11.x's Documents APIs and DynamoDB Mapper APIs in a
+> straight-forward manner. 
+
+> Option 2: A generic document database client that creates an
+> abstraction over all document databases, like DynamoDB and MongoDB.
+> This would simplify using multiple document databases in the same
+> application, and make it easier to migrate between the two.
+> Unfortunately as a result, it also wouldn't be a direct DynamoDB
+> experience.
+
+We requested that customers review these two options as well as a
+[prototype of option 1](prototype/option-1/sync/Prototype.java) and a
+[prototype of option 2](prototype/option-2/sync/Prototype.java), to let
+us know which they prefer.
+
+The following anecdotes are from this customer feedback:
+
+> If \[Amazon] can make something like https://serverless.com/ or
+> https://onnx.ai/ which free customers from vendor lock-in, that would
+> be a great Think Big & Customer Obsession idea. If \[Amazon] cannot,
+> I feel that somebody who is more vendor-neutral can make a better
+> mapper than \[Amazon].
+
+> Have you thought about contributing to projects which already exist,
+> like Spring Data? https://github.com/derjust/spring-data-dynamodb 
+
+> Both options would work well for us.
+
+> I think \[doing option 1 and then creating a Spring Data plugin] might
+> get adoption from a broader audience than option 2. It could be used
+> as a stepping stone to move to DynamoDB.
+
+> I believe Option 2 does not make much sense. It would make sense to me
+> to go for Option 1 and start a bounty program to implement a module to
+> popular data access abstraction libraries such as spring-data
+> mentioned above or GORM.
+
+> Maybe you could implement/support JNOSQL spec http://www.jnosql.org/
+
+**Decision**
+
+Based on customer feedback, it was decided to temporarily reject
+alternative solution 1, and build the proposed solution. At a later
+time, the SDK may build a Level 3 abstraction for DynamoDB or integrate
+with existing Java Level 3 abstractions like Spring Data, Hibernate OGM,
+and/or JNoSQL. This Level 3 abstraction will possibly leverage the Level
+2 solution "under the hood".
+
+## Links
 
 **[Features](features.md)** - The features intended for inclusion during
 and after the launch of the enhanced DynamoDB client.
@@ -55,26 +365,4 @@ to solicit feedback from customers on potential design directions.
   for DynamoDB mapper-equivalent functionality in 2.x.
 * [DynamoDB Document API Feature Request](https://github.com/aws/aws-sdk-java-v2/issues/36)
   \- A github issue for tracking customer feature requests and feedback
-  for DynamoDB document API-equivalent functionality in 2.x.
-
-# FAQ
-
-**Why not optimize for specific data access patterns, like relational
-data?**
-
-Some customers prefer to think about the DynamoDB-specific concepts of
-tables, queries, conditions and global secondary indices. These
-customers currently have trouble interacting with DynamoDB, because they
-have to perform a large amount of conversion between their Java types
-and DynamoDB types. 
-
-If we were to optimize this specific project for a particular access
-pattern, customers that have other access patterns would either be
-forced to modify their access pattern into the one we've optimized for,
-or they would need to use the generated client and handle their own Java
-type conversion.
-
-Instead, this project focuses on making integrating with DynamoDB easier
-to do in Java, and leaves the problems of "making storing time series
-data on AWS easy" or "making storing documents on AWS easy" to another
-project.
+  for DynamoDB document API-equivalent functionality in 2.x.
