chore(v2): Split powertools-idempotency module to sub-modules and add redis implementation #1513
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,7 +35,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| ... | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| <version>{{ powertools.version }}</version> | ||
| </dependency> | ||
| ... | ||
|
|
@@ -56,7 +56,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| <aspectLibraries> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| </aspectLibrary> | ||
| </aspectLibraries> | ||
| </configuration> | ||
|
|
@@ -80,7 +80,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| ... | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| <version>{{ powertools.version }}</version> | ||
| </dependency> | ||
| ... | ||
|
|
@@ -101,7 +101,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| <aspectLibraries> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| </aspectLibrary> | ||
| </aspectLibraries> | ||
| </configuration> | ||
|
|
@@ -131,7 +131,7 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| } | ||
|
|
||
| dependencies { | ||
| aspect 'software.amazon.lambda:powertools-idempotency-dynamodb:{{ powertools.version }}' | ||
| } | ||
|
|
||
| sourceCompatibility = 11 // or higher | ||
|
|
@@ -151,18 +151,20 @@ Depending on your version of Java (either Java 1.8 or 11+), the configuration sl | |
| } | ||
|
|
||
| dependencies { | ||
| aspect 'software.amazon.lambda:powertools-idempotency-dynamodb:{{ powertools.version }}' | ||
| } | ||
|
|
||
| sourceCompatibility = 1.8 | ||
| targetCompatibility = 1.8 | ||
| ``` | ||
|
|
||
| ### Required resources | ||
| Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your Lambda functions will need read and write access to it. | ||
| As of now, [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) and [Redis](https://redis.io/) are the supported persistnce layers. | ||
|
|
||
| #### Using Amazon DynamoDB | ||
|
|
||
| If you are using Amazon DynamoDB, you'll need to create a table. | ||
|
|
||
| **Default table configuration** | ||
|
|
||
|
|
@@ -215,12 +217,65 @@ Resources: | |
| see 1WCU and 1RCU. Review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/) to | ||
| estimate the cost. | ||
|
|
||
| #### Using Redis | ||
|
|
||
| ##### Redis resources | ||
|
|
||
| You need an existing Redis service before setting up Redis as the persistent storage layer provider. You can also use Redis compatible services like [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/) or [Amazon MemoryDB for Redis](https://aws.amazon.com/memorydb/) as persistent storage layer provider. | ||
|
|
||
| !!! tip "Tip:No existing Redis service?" | ||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| If you don't have an existing Redis service, we recommend using DynamoDB as persistent storage layer provider. DynamoDB does not require a VPC deployment and is easier to configure and operate. | ||
|
|
||
| If you want to connect to a Redis cluster instead of a Standalone server, you need to enable Redis cluster mode by setting an AWS Lambda | ||
| environment variable `REDIS_CLUSTER_MODE` to `true` | ||
| In the following example, you can see a SAM template for deploying an AWS Lambda function by specifying the required environment variable. | ||
|
|
||
| !!! warning "Warning: Large responses with Redis persistence layer" | ||
| When using this utility with Redis your function's responses must be smaller than 512MB. | ||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Persisting larger items might cause exceptions. | ||
|
|
||
| ```yaml hl_lines="9" title="AWS Serverless Application Model (SAM) example" | ||
| Resources: | ||
| IdempotencyFunction: | ||
| Type: AWS::Serverless::Function | ||
| Properties: | ||
| CodeUri: Function | ||
| Handler: helloworld.App::handleRequest | ||
| Environment: | ||
| Variables: | ||
| REDIS_CLUSTER_MODE: "true" | ||
| ``` | ||
|
|
||
| ##### VPC Access | ||
eldimi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Your AWS Lambda Function must be able to reach the Redis endpoint before using it for idempotency persistent storage layer. In most cases you will need to [configure VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html) for your AWS Lambda Function. Using a public accessible Redis is not recommended. | ||
|
|
||
| !!! tip "Amazon ElastiCache for Redis as persistent storage layer provider" | ||
| If you intend to use Amazon ElastiCache for Redis for idempotency persistent storage layer, you can also consult [this AWS tutorial](https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html). | ||
|
|
||
| ```yaml hl_lines="7-12" title="AWS Serverless Application Model (SAM) example" | ||
| Resources: | ||
| IdempotencyFunction: | ||
| Type: AWS::Serverless::Function | ||
| Properties: | ||
| CodeUri: Function | ||
| Handler: helloworld.App::handleRequest | ||
| VpcConfig: # (1)! | ||
| SecurityGroupIds: # (2)! | ||
| - sg-{your_sg_id} | ||
|
There was a problem hiding this comment. probably deserve more explanations: SG of elasticache / SG of the function? what would be the link between the two SGs (port 6379, 6380 ?) |
||
| SubnetIds: # (3)! | ||
| - subnet-{your_subnet_id_1} | ||
| - subnet-{your_subnet_id_2} | ||
| ``` | ||
| 1. Replace the Security Group ID and Subnet ID to match your Redis' VPC setting. | ||
| 2. The security group ID or IDs of the VPC where the Redis is deployed. | ||
| 3. The subnet IDs of the VPC where Redis is deployed. | ||
|
|
||
| ### Idempotent annotation | ||
|
|
||
| You can quickly start by initializing the persistence store used (e.g. `DynamoDBPersistenceStore` or `RedisPersistenceStore`) and using it with the `@Idempotent` annotation on your Lambda handler. | ||
|
|
||
| !!! warning "Important" | ||
| Initialization and configuration of the persistence store must be performed outside the handler, preferably in the constructor. | ||
|
|
||
| === "App.java" | ||
|
|
||
|
|
@@ -635,6 +690,29 @@ When using DynamoDB as a persistence layer, you can alter the attribute names by | |
| | **SortKeyAttr** | | | Sort key of the table (if table is configured with a sort key). | | ||
| | **StaticPkValue** | | `idempotency#{LAMBDA_FUNCTION_NAME}` | Static value to use as the partition key. Only used when **SortKeyAttr** is set. | | ||
|
|
||
| #### RedisPersistenceStore | ||
|
|
||
| The redis persistence store has as a prerequisite to install a Redis datastore(https://redis.io/docs/about/) in either Standalone or Cluster mode. | ||
|
|
||
| We are using [Redis hashes](https://redis.io/docs/data-types/hashes/) to store the idempotency fields and values. | ||
| There are some predefined fields that you can see listed in the following table. The predefined fields have some default values. | ||
|
|
||
|
|
||
| You can alter the field names by passing these parameters when initializing the persistence layer: | ||
|
|
||
| | Parameter | Required | Default | Description | | ||
| |--------------------|----------|--------------------------------------|--------------------------------------------------------------------------------------------------------| | ||
| | **KeyPrefixName** | Y | `idempotency` | The redis hash key prefix | | ||
| | **KeyAttr** | Y | `id` | The redis hash key field name | | ||
| | **ExpiryAttr** | | `expiration` | Unix timestamp of when record expires | | ||
| | **StatusAttr** | | `status` | Stores status of the Lambda execution during and after invocation | | ||
| | **DataAttr** | | `data` | Stores results of successfully idempotent methods | | ||
| | **ValidationAttr** | | `validation` | Hashed representation of the parts of the event used for validation | | ||
|
|
||
|
|
||
| !!! Tip "Tip: You can share the same prefix and key for all functions" | ||
| You can reuse the same prefix and key to store idempotency state. We add your function name in addition to the idempotency key as a hash key. | ||
|
|
||
| ## Advanced | ||
|
|
||
| ### Customizing the default behavior | ||
|
|
@@ -884,6 +962,49 @@ When creating the `DynamoDBPersistenceStore`, you can set a custom [`DynamoDbCli | |
| .build(); | ||
| ``` | ||
|
|
||
| ### Customizing Redis client | ||
|
|
||
| The `RedisPersistenceStore` uses the [`JedisPooled`](https://www.javadoc.io/doc/redis.clients/jedis/latest/redis/clients/jedis/JedisPooled.html) java client to connect to the Redis standalone server or the [`JedisCluster`](https://javadoc.io/doc/redis.clients/jedis/4.0.0/redis/clients/jedis/JedisCluster.html) to connect to the Redis cluster. | ||
| When creating the `RedisPersistenceStore`, you can set a custom Jedis client: | ||
|
|
||
| === "Custom JedisPooled with connection timeout" | ||
|
|
||
| ```java hl_lines="2-11 13 18" | ||
| public App() { | ||
| JedisConfig jedisConfig = JedisConfig.Builder.builder() | ||
| .withHost("redisHost") | ||
| .withPort("redisPort") | ||
| .withJedisClientConfig(DefaultJedisClientConfig.builder() | ||
| .user("user") | ||
| .password("secret") // leverage parameters-secrets module to retrieve this from Secrets Manager | ||
| .ssl(true) | ||
| .database(1) | ||
| .connectionTimeoutMillis(3000) | ||
| .build()) | ||
| .build(); | ||
|
|
||
| JedisPooled jedisPooled = new JedisPooled(new HostAndPort("host",6789), jedisConfig); | ||
|
|
||
| Idempotency.config().withPersistenceStore( | ||
| RedisPersistenceStore.builder() | ||
| .withKeyPrefixName("items-idempotency") | ||
| .withJedisClient(jedisPooled) | ||
| .build() | ||
| ).configure(); | ||
| } | ||
| ``` | ||
|
|
||
| !!! info "Default configuration is the following:" | ||
|
There was a problem hiding this comment. What do you mean by "default configuration", it is what we use as default ? If that's the case, does it make sense? Not sure anyone is using the user "default" with no password... I think we should ask for all the required fields to create a JedisClient ourself:
@scottgerring wdyt ? There was a problem hiding this comment. Default here means, as provided by the jedis client default config. Actually I have to update the powertools documentation, since user/pass, by default, in JedisClient are null. There was a problem hiding this comment. Found also this extract here from redis.conf
|
||
|
|
||
| ```java | ||
| DefaultJedisClientConfig.builder() | ||
| .user(null) | ||
| .password(null) | ||
| .ssl(false) | ||
| .database(0) | ||
| .build(); | ||
| ``` | ||
|
|
||
| ### Using a DynamoDB table with a composite primary key | ||
|
|
||
| When using a composite primary key table (hash+range key), use `SortKeyAttr` parameter when initializing your persistence store. | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -421,7 +421,6 @@ | |
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <goals> | ||
| <goal>compile</goal> | ||
| <goal>test-compile</goal> | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <parent> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>e2e-test-handlers-parent</artifactId> | ||
| <version>1.0.0</version> | ||
| </parent> | ||
|
|
||
| <artifactId>e2e-test-handler-idempotency-dynamodb</artifactId> | ||
| <packaging>jar</packaging> | ||
| <name>A Lambda function using Powertools for AWS Lambda (Java) idempotency</name> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-dynamodb</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-logging</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.logging.log4j</groupId> | ||
| <artifactId>log4j-slf4j2-impl</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>com.amazonaws</groupId> | ||
| <artifactId>aws-lambda-java-events</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>org.aspectj</groupId> | ||
| <artifactId>aspectjrt</artifactId> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| <build> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>dev.aspectj</groupId> | ||
| <artifactId>aspectj-maven-plugin</artifactId> | ||
| <configuration> | ||
| <source>${maven.compiler.source}</source> | ||
| <target>${maven.compiler.target}</target> | ||
| <complianceLevel>${maven.compiler.target}</complianceLevel> | ||
| <aspectLibraries> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-idempotency-core</artifactId> | ||
| </aspectLibrary> | ||
| <aspectLibrary> | ||
| <groupId>software.amazon.lambda</groupId> | ||
| <artifactId>powertools-logging</artifactId> | ||
| </aspectLibrary> | ||
| </aspectLibraries> | ||
| </configuration> | ||
| <executions> | ||
| <execution> | ||
| <goals> | ||
| <goal>compile</goal> | ||
| </goals> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-shade-plugin</artifactId> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.