Skip to content

chore: Sync main and develop #526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Powertools for AWS Lambda (.NET) provides three core utilities:

* **[Parameters](https://docs.powertools.aws.dev/lambda/dotnet/utilities/parameters/)** - provides high-level functionality to retrieve one or multiple parameter values from [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html), [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), or [Amazon DynamoDB](https://aws.amazon.com/dynamodb/). We also provide extensibility to bring your own providers.

* **[Idempotency (developer preview)](https://docs.powertools.aws.dev/lambda/dotnet/utilities/idempotency/)** - The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.
* **[Idempotency](https://docs.powertools.aws.dev/lambda/dotnet/utilities/idempotency/)** - The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.

* **[Batch Processing (developer preview)](https://docs.powertools.aws.dev/lambda/dotnet/utilities/batch-processing/)** - The batch processing utility handles partial failures when processing batches from Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams.

Expand Down
18 changes: 11 additions & 7 deletions docs/core/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,14 +72,19 @@ Here is an example using the AWS SAM [Globals section](https://docs.aws.amazon.c

### Using AWS Lambda Advanced Logging Controls (ALC)

With [AWS Lambda Advanced Logging Controls (ALC)](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-advanced), you can control the output format of your logs as either TEXT or JSON and specify the minimum accepted log level for your application. Regardless of the output format setting in Lambda, Powertools for AWS Lambda will always output JSON formatted logging messages.
!!! question "When is it useful?"
When you want to set a logging policy to drop informational or verbose logs for one or all AWS Lambda functions, regardless of runtime and logger used.

When you have this feature enabled, log messages that don’t meet the configured log level are discarded by Lambda. For example, if you set the minimum log level to WARN, you will only receive WARN and ERROR messages in your AWS CloudWatch Logs, all other log levels will be discarded by Lambda.
With [AWS Lambda Advanced Logging Controls (ALC)](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-advanced){target="_blank"}, you can enforce a minimum log level that Lambda will accept from your application code.

When enabled, you should keep `Logger` and ALC log level in sync to avoid data loss.

!!! warning "When using AWS Lambda Advanced Logging Controls (ALC)"
- When Powertools Logger output is set to `PascalCase` **`Level`** property name will be replaced by **`LogLevel`** as a property name.
- ALC takes precedence over **`POWERTOOLS_LOG_LEVEL`** and when setting it in code using **`[Logging(LogLevel = )]`**

Here's a sequence diagram to demonstrate how ALC will drop both `Information` and `Debug` logs emitted from `Logger`, when ALC log level is stricter than `Logger`.

```mermaid
sequenceDiagram
title Lambda ALC allows WARN logs only
Expand All @@ -88,6 +93,7 @@ sequenceDiagram
participant Application Logger

Note over Lambda service: AWS_LAMBDA_LOG_LEVEL="WARN"
Note over Application Logger: POWERTOOLS_LOG_LEVEL="DEBUG"
Lambda service->>Lambda function: Invoke (event)
Lambda function->>Lambda function: Calls handler
Lambda function->>Application Logger: Logger.Warning("Something happened")
Expand All @@ -99,20 +105,18 @@ sequenceDiagram
Lambda service->>CloudWatch Logs: Ingest error logs
```

Logger will automatically listen for the AWS_LAMBDA_LOG_FORMAT and AWS_LAMBDA_LOG_LEVEL environment variables, and change behaviour if they’re found to ensure as much compatibility as possible.

**Priority of log level settings in Powertools for AWS Lambda**

When the Advanced Logging Controls feature is enabled, we are unable to increase the minimum log level below the AWS_LAMBDA_LOG_LEVEL environment variable value, see [AWS Lambda service documentation](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-log-level) for more details.

We prioritise log level settings in this order:

1. AWS_LAMBDA_LOG_LEVEL environment variable
2. Setting the log level in code using `[Logging(LogLevel = )]`
3. POWERTOOLS_LOG_LEVEL environment variable

In the event you have set POWERTOOLS_LOG_LEVEL to a level lower than the ACL setting, Powertools for AWS Lambda will output a warning log message informing you that your messages will be discarded by Lambda.
If you set `Logger` level lower than ALC, we will emit a warning informing you that your messages will be discarded by Lambda.

> **NOTE**
> With ALC enabled, we are unable to increase the minimum log level below the `AWS_LAMBDA_LOG_LEVEL` environment variable value, see [AWS Lambda service documentation](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html#monitoring-cloudwatchlogs-log-level){target="_blank"} for more details.

## Standard structured keys

Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Core utilities such as Tracing, Logging, and Metrics will be available across al
[Logger](./core/logging.md) | Structured logging made easier, and decorator to enrich structured logging with key Lambda context details
[Metrics](./core/metrics.md) | Custom AWS metrics created asynchronously via CloudWatch Embedded Metric Format (EMF)
[Parameters](./utilities/parameters/) | provides high-level functionality to retrieve one or multiple parameter values from [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html){target="_blank"}, [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/){target="_blank"}, or [Amazon DynamoDB](https://aws.amazon.com/dynamodb/){target="_blank"}. We also provide extensibility to bring your own providers.
[Idempotency (developer preview)](./utilities/idempotency/) | The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.
[Idempotency](./utilities/idempotency/) | The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.
[Batch Processing (developer preview)](./utilities/batch-processing/) | The batch processing utility handles partial failures when processing batches from Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams.

## Install
Expand Down Expand Up @@ -128,4 +128,4 @@ These are our core principles to guide our decision making.
* **Keep it lean**. Additional dependencies are carefully considered for security and ease of maintenance, and prevent negatively impacting startup time.
* **We strive for backwards compatibility**. New features and changes should keep backwards compatibility. If a breaking change cannot be avoided, the deprecation and migration process should be clearly defined.
* **We work backwards from the community**. We aim to strike a balance of what would work best for 80% of customers. Emerging practices are considered and discussed via Requests for Comment (RFCs)
* **Idiomatic**. Utilities follow programming language idioms and language-specific best practices.
* **Idiomatic**. Utilities follow programming language idioms and language-specific best practices.
2 changes: 0 additions & 2 deletions docs/utilities/idempotency.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@
title: Idempotency
description: Utility
---
???+ warning
**This utility is currently in developer preview** and is intended strictly for feedback and testing purposes **and not for production workloads**. The version and all future versions tagged with the `-preview` suffix should be treated as not stable. Until this utility is [General Availability](https://github.com/aws-powertools/powertools-lambda-dotnet/milestone/3) we may introduce significant breaking changes and improvements in response to customers feedback.

<!-- markdownlint-disable MD013 -->

Expand Down
2 changes: 1 addition & 1 deletion examples/BatchProcessing/src/HelloWorld/HelloWorld.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="AWS.Lambda.Powertools.BatchProcessing" Version="0.0.1-preview" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="7.0.0" />
</ItemGroup>
</Project>
2 changes: 1 addition & 1 deletion examples/Idempotency/src/HelloWorld/HelloWorld.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,6 @@
<PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.6.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="AWS.Lambda.Powertools.Idempotency" Version="0.2.2-preview" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
</ItemGroup>
</Project>
2 changes: 1 addition & 1 deletion examples/Logging/src/HelloWorld/HelloWorld.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.6.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
<PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.201.13" />
</ItemGroup>
</Project>
4 changes: 2 additions & 2 deletions examples/Metrics/src/HelloWorld/HelloWorld.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.6.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Metrics" Version="1.4.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
<PackageReference Include="AWS.Lambda.Powertools.Metrics" Version="1.5.3" />
<PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.201.13" />
</ItemGroup>
</Project>
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.AspNetCoreServer" Version="8.1.0" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Metrics" Version="1.4.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
<PackageReference Include="AWS.Lambda.Powertools.Metrics" Version="1.5.3" />
<PackageReference Include="AWS.Lambda.Powertools.Tracing" Version="1.3.2" />
</ItemGroup>
</Project>
2 changes: 1 addition & 1 deletion examples/Tracing/src/HelloWorld/HelloWorld.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.6.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.3.3" />
<PackageReference Include="AWS.Lambda.Powertools.Logging" Version="1.4.4" />
<PackageReference Include="AWS.Lambda.Powertools.Tracing" Version="1.3.2" />
<PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.201.13" />
</ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/apache2.0
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/

using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Text.Json;
using System.Threading.Tasks;
using Amazon.DynamoDBv2;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;

namespace AWS.Lambda.Powertools.Idempotency.Tests.Handlers;

public class IdempotencyFunctionMethodDecorated
{
public bool MethodCalled;

public IdempotencyFunctionMethodDecorated(AmazonDynamoDBClient client)
{
Idempotency.Configure(builder =>
builder
.UseDynamoDb(storeBuilder =>
storeBuilder
.WithTableName("idempotency_table")
.WithDynamoDBClient(client)
));
}


public async Task<APIGatewayProxyResponse> Handle(APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
{
Idempotency.RegisterLambdaContext(context);
var result= await InternalFunctionHandler(apigProxyEvent);

return result;
}

private async Task<APIGatewayProxyResponse> InternalFunctionHandler(APIGatewayProxyRequest apigProxyEvent)
{
Dictionary<string, string> headers = new()
{
{"Content-Type", "application/json"},
{"Access-Control-Allow-Origin", "*"},
{"Access-Control-Allow-Methods", "GET, OPTIONS"},
{"Access-Control-Allow-Headers", "*"}
};

try
{
var address = JsonDocument.Parse(apigProxyEvent.Body).RootElement.GetProperty("address").GetString();
var pageContents = await GetPageContents(address);
var output = $"{{ \"message\": \"hello world\", \"location\": \"{pageContents}\" }}";

return new APIGatewayProxyResponse
{
Body = output,
StatusCode = 200,
Headers = headers
};

}
catch (IOException)
{
return new APIGatewayProxyResponse
{
Body = "{}",
StatusCode = 500,
Headers = headers
};
}
}

[Idempotent]
private async Task<string> GetPageContents(string address)
{
MethodCalled = true;

var client = new HttpClient();
using var response = await client.GetAsync(address);
using var content = response.Content;
var pageContent = await content.ReadAsStringAsync();

return pageContent;
}
}
Original file line number Diff line number Diff line change
@@ -1,24 +1,27 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
*
* Licensed under the Apache License, Version 2.0 (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
*
* http://aws.amazon.com/apache2.0
*
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/

using System;
using System.Collections.Generic;
using System.IO;
using System.Text.Json;
using System.Threading.Tasks;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.Model;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.TestUtilities;
using AWS.Lambda.Powertools.Idempotency.Tests.Handlers;
using AWS.Lambda.Powertools.Idempotency.Tests.Persistence;
using FluentAssertions;
Expand Down Expand Up @@ -66,5 +69,57 @@ public async Task EndToEndTest()
TableName = _tableName
});
scanResponse.Count.Should().Be(1);

// delete row dynamo
var key = new Dictionary<string, AttributeValue>
{
["id"] = new AttributeValue { S = "testFunction.GetPageContents#ff323c6f0c5ceb97eed49121babcec0f" }
};
await _client.DeleteItemAsync(new DeleteItemRequest{TableName = _tableName, Key = key});
}

[Fact]
[Trait("Category", "Integration")]
public async Task EndToEndTestMethod()
{
var function = new IdempotencyFunctionMethodDecorated(_client);

var options = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true
};

var context = new TestLambdaContext
{
RemainingTime = TimeSpan.FromSeconds(30)
};

var request = JsonSerializer.Deserialize<APIGatewayProxyRequest>(await File.ReadAllTextAsync("./resources/apigw_event2.json"),options);

var response = await function.Handle(request, context);
function.MethodCalled.Should().BeTrue();

function.MethodCalled = false;

var response2 = await function.Handle(request, context);
function.MethodCalled.Should().BeFalse();

// Assert
JsonSerializer.Serialize(response).Should().Be(JsonSerializer.Serialize(response2));
response.Body.Should().Contain("hello world");
response2.Body.Should().Contain("hello world");

var scanResponse = await _client.ScanAsync(new ScanRequest
{
TableName = _tableName
});
scanResponse.Count.Should().Be(1);

// delete row dynamo
var key = new Dictionary<string, AttributeValue>
{
["id"] = new AttributeValue { S = "testFunction.GetPageContents#ff323c6f0c5ceb97eed49121babcec0f" }
};
await _client.DeleteItemAsync(new DeleteItemRequest{TableName = _tableName, Key = key});
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ public void Handle_WhenIdempotencyOnSubMethodAnnotated_AndSecondCall_AndNotExpir
// Arrange
var store = Substitute.For<BasePersistenceStore>();
store.SaveInProgress(Arg.Any<JsonDocument>(), Arg.Any<DateTimeOffset>(), Arg.Any<double>())
.Throws(new IdempotencyItemAlreadyExistsException());
.Returns(_ => throw new IdempotencyItemAlreadyExistsException());

Idempotency.Configure(builder => builder.WithPersistenceStore(store));

Expand All @@ -327,7 +327,7 @@ public void Handle_WhenIdempotencyOnSubMethodAnnotated_AndSecondCall_AndNotExpir
.Returns(record);

// Act
var function = new IdempotencyInternalFunction(false);
var function = new IdempotencyInternalFunction(true);
Basket resultBasket = function.HandleRequest(product, new TestLambdaContext());

// assert
Expand Down
2 changes: 1 addition & 1 deletion version.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
},
"Utilities": {
"Parameters": "1.1.2",
"Idempotency": "0.2.2-preview",
"Idempotency": "1.0.0",
"BatchProcessing": "0.0.1-preview"
}
}