Skip to content

chore(middleware-content-length): set transfer-encoding if content-length is not set #3403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 9, 2022

Conversation

trivikr
Copy link
Member

@trivikr trivikr commented Mar 9, 2022

Issue

Refs: #3400

Description

Sets transfer-encoding if content-length is not set.
Refs: #3400 (comment)

Testing

  • Integration testing
  • Unit tests are not present for the middleware, and should be added in a separate PR.

Verified that transfer-encoding header is set to chunked in the following test case when we attempt to send ReadableStream to PutObject:

import { S3 } from "../aws-sdk-js-v3/clients/client-s3/dist-cjs/index.js";
import { Readable } from "stream";

const region = "us-west-2";
const client = new S3({ region });

const readableStream = new Readable();

const mockDataChunks = ["Hello", "World"];
setTimeout(() => {
  mockDataChunks.forEach((chunk) => readableStream.emit("data", chunk));
  readableStream.emit("end");
}, 1000);

const Bucket = "aws-sdk-js-trivikr";
const Key = "hello-world.txt";
const Body = readableStream;

try {
  await client.putObject({ Bucket, Key, Body });
} catch (error) {
  console.log({ error });
}

Irrespective of whether Transfer-Encoding is set (in PR) or not (in v3.53.1 with flexible checksums, or v3.52.0 prior to flexible checksums), the code fails with error:

NotImplemented: A header you provided implies functionality that is not implemented
      at deserializeAws_restXmlPutObjectCommandError (/local/home/trivikr/workspace/aws-sdk-js-v3/clients/client-s3/dist-cjs/protocols/Aws_restXml.js:8037:24)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async /local/home/trivikr/workspace/aws-sdk-js-v3/packages/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
      at async /local/home/trivikr/workspace/aws-sdk-js-v3/packages/middleware-signing/dist-cjs/middleware.js:11:20
      at async StandardRetryStrategy.retry (/local/home/trivikr/workspace/aws-sdk-js-v3/packages/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
      at async /local/home/trivikr/workspace/aws-sdk-js-v3/packages/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:56:20
      at async /local/home/trivikr/workspace/aws-sdk-js-v3/packages/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
      at async file:///local/home/trivikr/workspace/temp/putObject.fd.mjs:20:3 {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 501,
      requestId: undefined,
      extendedRequestId: 'YUlBcbnai0WfSRASq95c8Kh06Jg1qFmJL1RJLLHFFHpQIY3pD8+LZry4nv1U6bOYlkFK3vcdYAk=',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    Code: 'NotImplemented',
    Header: 'Transfer-Encoding',
    RequestId: 'FFVSGD9GE62NHY9T',
    HostId: 'YUlBcbnai0WfSRASq95c8Kh06Jg1qFmJL1RJLLHFFHpQIY3pD8+LZry4nv1U6bOYlkFK3vcdYAk=',
    '$response': HttpResponse {
      statusCode: 501,
      headers: [Object],
      body: [IncomingMessage]
    }
  }

The error should be discussed separately if needed. We're following RFC7230 spec, and the behavior in AWS SDK for Python.

Additional context

This PR will be rebased and made ready once #3401 is merged. Ready


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@trivikr trivikr force-pushed the mw-content-length-transfer-encoding branch from 3fcd3c8 to dec9143 Compare March 9, 2022 01:20
@trivikr trivikr marked this pull request as ready for review March 9, 2022 01:21
@trivikr trivikr requested a review from a team as a code owner March 9, 2022 01:21
@trivikr trivikr merged commit 7e4abf7 into aws:main Mar 9, 2022
@trivikr trivikr deleted the mw-content-length-transfer-encoding branch March 9, 2022 17:01
@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants