Don't read full file if range header is present #876

JonnyBurger · 2021-04-15T11:27:06Z

This PR contains a:

Motivation / Use-Case

I am reading in a 2GB file for Remotion. This is a framework for making videos in React, and the end product is not shipped to a website, but written to an MP4 file. Therefore it's not a problem that we import these large files in Webpack.

However, when the browser is loading a video, and we seek forward, then a range header is sent to Webpack Dev Server. Unfortunately it is slow, because WDS is still loading the full file. I adapted the logic from serve-handler instead so instead of reading the full 2GB file synchronously, it uses a Read Stream.

Breaking Changes

Aiming to have no breaking changes.

Additional Info

Will try to finish this PR when I get time. Might take a while!

package.json

alexander-akait

Also please add tests

JonnyBurger · 2021-06-06T12:19:13Z

@alexander-akait

I finally got around to making this PR, all the tests are passing now!
There are already tests for the range header, I added some more assertions that the actual response has the correct length and content.

codecov · 2021-06-06T12:29:33Z

Codecov Report

Merging #876 (bc4955e) into master (d50e308) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head bc4955e differs from pull request most recent head 043da8a. Consider uploading reports for the commit 043da8a to get more accurate results

@@           Coverage Diff           @@
##           master     #876   +/-   ##
=======================================
  Coverage   97.90%   97.91%           
=======================================
  Files          10       10           
  Lines         287      288    +1     
  Branches      100      102    +2     
=======================================
+ Hits          281      282    +1     
  Misses          6        6

Impacted Files	Coverage Δ
src/middleware.js	`96.15% <100.00%> (+0.15%)`	⬆️
src/utils/handleRangeHeaders.js	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d50e308...043da8a. Read the comment docs.

fix length mismatch should not close before fix content length remove headers add res.send back again Update middleware.js make all tests but 1 pass clarify that multiple ranges are not supported size -> fileSize and resolve warning clean up contentRes -> ranges minimize diff fix content length bug add assertion for response length add more assertions validate the correct content Update handleRangeHeaders.test.js.snap.webpack4 fix: return on error

JonnyBurger · 2021-08-11T09:26:38Z

@alexander-akait I think code coverage decrease should be fixed. Can you approve the CI runs? If everything goes green, it's ready for review.

Bumps [prettier](https://github.com/prettier/prettier) from 2.3.0 to 2.3.1. - [Release notes](https://github.com/prettier/prettier/releases) - [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md) - [Commits](prettier/prettier@2.3.0...2.3.1) --- updated-dependencies: - dependency-name: prettier dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [eslint](https://github.com/eslint/eslint) from 7.27.0 to 7.28.0. - [Release notes](https://github.com/eslint/eslint/releases) - [Changelog](https://github.com/eslint/eslint/blob/master/CHANGELOG.md) - [Commits](eslint/eslint@v7.27.0...v7.28.0) --- updated-dependencies: - dependency-name: eslint dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [execa](https://github.com/sindresorhus/execa) from 5.1.0 to 5.1.1. - [Release notes](https://github.com/sindresorhus/execa/releases) - [Commits](sindresorhus/execa@v5.1.0...v5.1.1) --- updated-dependencies: - dependency-name: execa dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jest](https://github.com/facebook/jest) from 27.0.3 to 27.0.4. - [Release notes](https://github.com/facebook/jest/releases) - [Changelog](https://github.com/facebook/jest/blob/master/CHANGELOG.md) - [Commits](jestjs/jest@v27.0.3...v27.0.4) --- updated-dependencies: - dependency-name: jest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…k#939)

Bumps [chokidar](https://github.com/paulmillr/chokidar) from 3.5.1 to 3.5.2. - [Release notes](https://github.com/paulmillr/chokidar/releases) - [Commits](paulmillr/chokidar@3.5.1...3.5.2) --- updated-dependencies: - dependency-name: chokidar dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.39.0 to 5.39.1. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.39.0...v5.39.1) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [@babel/cli](https://github.com/babel/babel/tree/HEAD/packages/babel-cli) from 7.14.3 to 7.14.5. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.14.5/packages/babel-cli) --- updated-dependencies: - dependency-name: "@babel/cli" dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander Akait <[email protected]>

Bumps [babel-jest](https://github.com/facebook/jest/tree/HEAD/packages/babel-jest) from 27.0.2 to 27.0.5. - [Release notes](https://github.com/facebook/jest/releases) - [Changelog](https://github.com/facebook/jest/blob/master/CHANGELOG.md) - [Commits](https://github.com/facebook/jest/commits/v27.0.5/packages/babel-jest) --- updated-dependencies: - dependency-name: babel-jest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jest](https://github.com/facebook/jest) from 27.0.4 to 27.0.5. - [Release notes](https://github.com/facebook/jest/releases) - [Changelog](https://github.com/facebook/jest/blob/master/CHANGELOG.md) - [Commits](jestjs/jest@v27.0.4...v27.0.5) --- updated-dependencies: - dependency-name: jest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [prettier](https://github.com/prettier/prettier) from 2.3.1 to 2.3.2. - [Release notes](https://github.com/prettier/prettier/releases) - [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md) - [Commits](prettier/prettier@2.3.1...2.3.2) --- updated-dependencies: - dependency-name: prettier dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [jest](https://github.com/facebook/jest) from 27.0.5 to 27.0.6. - [Release notes](https://github.com/facebook/jest/releases) - [Changelog](https://github.com/facebook/jest/blob/master/CHANGELOG.md) - [Commits](jestjs/jest@v27.0.5...v27.0.6) --- updated-dependencies: - dependency-name: jest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [del-cli](https://github.com/sindresorhus/del-cli) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/sindresorhus/del-cli/releases) - [Commits](sindresorhus/del-cli@v4.0.0...v4.0.1) --- updated-dependencies: - dependency-name: del-cli dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.44.0 to 5.45.1. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.44.0...v5.45.1) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [eslint](https://github.com/eslint/eslint) from 7.30.0 to 7.31.0. - [Release notes](https://github.com/eslint/eslint/releases) - [Changelog](https://github.com/eslint/eslint/blob/master/CHANGELOG.md) - [Commits](eslint/eslint@v7.30.0...v7.31.0) --- updated-dependencies: - dependency-name: eslint dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [schema-utils](https://github.com/webpack/schema-utils) from 3.1.0 to 3.1.1. - [Release notes](https://github.com/webpack/schema-utils/releases) - [Changelog](https://github.com/webpack/schema-utils/blob/master/CHANGELOG.md) - [Commits](webpack/schema-utils@v3.1.0...v3.1.1) --- updated-dependencies: - dependency-name: schema-utils dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [supertest](https://github.com/visionmedia/supertest) from 6.1.3 to 6.1.4. - [Release notes](https://github.com/visionmedia/supertest/releases) - [Commits](ladjs/supertest@v6.1.3...v6.1.4) --- updated-dependencies: - dependency-name: supertest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.45.1 to 5.46.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.45.1...v5.46.0) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [lint-staged](https://github.com/okonet/lint-staged) from 11.0.1 to 11.1.0. - [Release notes](https://github.com/okonet/lint-staged/releases) - [Commits](lint-staged/lint-staged@v11.0.1...v11.1.0) --- updated-dependencies: - dependency-name: lint-staged dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [lint-staged](https://github.com/okonet/lint-staged) from 11.1.0 to 11.1.1. - [Release notes](https://github.com/okonet/lint-staged/releases) - [Commits](lint-staged/lint-staged@v11.1.0...v11.1.1) --- updated-dependencies: - dependency-name: lint-staged dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [mime-types](https://github.com/jshttp/mime-types) from 2.1.31 to 2.1.32. - [Release notes](https://github.com/jshttp/mime-types/releases) - [Changelog](https://github.com/jshttp/mime-types/blob/master/HISTORY.md) - [Commits](jshttp/mime-types@2.1.31...2.1.32) --- updated-dependencies: - dependency-name: mime-types dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.46.0 to 5.47.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.46.0...v5.47.0) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.47.0 to 5.47.1. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.47.0...v5.47.1) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [eslint](https://github.com/eslint/eslint) from 7.31.0 to 7.32.0. - [Release notes](https://github.com/eslint/eslint/releases) - [Changelog](https://github.com/eslint/eslint/blob/master/CHANGELOG.md) - [Commits](eslint/eslint@v7.31.0...v7.32.0) --- updated-dependencies: - dependency-name: eslint dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.47.1 to 5.48.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.47.1...v5.48.0) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [webpack](https://github.com/webpack/webpack) from 5.48.0 to 5.49.0. - [Release notes](https://github.com/webpack/webpack/releases) - [Commits](webpack/webpack@v5.48.0...v5.49.0) --- updated-dependencies: - dependency-name: webpack dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…pack#997)

Bumps [colorette](https://github.com/jorgebucaran/colorette) from 1.2.2 to 1.3.0. - [Release notes](https://github.com/jorgebucaran/colorette/releases) - [Commits](jorgebucaran/colorette@1.2.2...1.3.0) --- updated-dependencies: - dependency-name: colorette dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

JonnyBurger · 2021-08-11T09:51:48Z

I could not figure out how to correctly reformat the commit messages, I made a mess and started a clean PR here: #1002

alexander-akait reviewed Apr 15, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

alexander-akait requested changes Apr 15, 2021

View reviewed changes

JonnyBurger mentioned this pull request Apr 20, 2021

Move webpack-dev-middleware changes upstream remotion-dev/remotion#283

Closed

JonnyBurger force-pushed the master branch from 056ea81 to 952e7e3 Compare June 6, 2021 11:13

JonnyBurger marked this pull request as ready for review June 6, 2021 11:49

JonnyBurger requested review from hiroppy and snitin315 as code owners June 6, 2021 11:49

JonnyBurger force-pushed the master branch from afdac15 to a5ce745 Compare June 6, 2021 12:43

should not throw error while reading stream

544f52a

JonnyBurger requested a review from alexander-akait August 11, 2021 09:25

dependabot bot and others added 16 commits August 11, 2021 11:43

chore(deps-dev): bump @babel/core from 7.14.3 to 7.14.5 (webpack#937)

7cbdf2d

chore(deps-dev): bump @babel/preset-env from 7.14.4 to 7.14.5 (webpac…

a2e8bfb

…k#939)

chore: update deps (webpack#942)

7a9dda4

chore(deps-dev): bump webpack from 5.39.1 to 5.40.0 (webpack#947)

700b4e2

chore: update deps (webpack#949)

d466d2c

dependabot bot and others added 26 commits August 11, 2021 11:43

chore(deps-dev): bump @babel/preset-env

20971b1

ci: setup npm cache (webpack#983)

928c475

chore(deps-dev): bump @babel/core from

9e9eb57

chore(deps-dev): bump @babel/cli from

f7c03c1

chore(deps-dev): bump @babel/preset-env

8e001c0

chore: update commitlint action in CI (webpack#993)

5b9889c

chore(deps-dev): bump @babel/core

fca2070

chore(deps-dev): bump @babel/preset-env

442d5cf

ci: update codecov-action (webpack#996)

f1da1f3

chore(deps-dev): bump lint-staged from 11.1.1 to 11.1.2 (webpack#998)

6b2d16c

chore(deps-dev): bump eslint-plugin-import from 2.23.4 to 2.24.0 (web…

dc62b8e

…pack#997)

JonnyBurger force-pushed the master branch from bc4955e to 043da8a Compare August 11, 2021 09:44

JonnyBurger closed this Aug 11, 2021

terrorizer1980 mentioned this pull request Mar 14, 2023

[Snyk] Upgrade memfs from 3.4.0 to 3.4.13 Exhorder6/webpack-dev-middleware#6

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Don't read full file if range header is present #876

Don't read full file if range header is present #876

Uh oh!

JonnyBurger commented Apr 15, 2021 •

edited

Loading

Uh oh!

Uh oh!

alexander-akait left a comment

Uh oh!

JonnyBurger commented Jun 6, 2021

Uh oh!

codecov bot commented Jun 6, 2021 •

edited

Loading

Uh oh!

JonnyBurger commented Aug 11, 2021

Uh oh!

JonnyBurger commented Aug 11, 2021

Uh oh!

Uh oh!

Uh oh!

Don't read full file if range header is present #876

Don't read full file if range header is present #876

Uh oh!

Conversation

JonnyBurger commented Apr 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation / Use-Case

Breaking Changes

Additional Info

Uh oh!

Uh oh!

alexander-akait left a comment

Choose a reason for hiding this comment

Uh oh!

JonnyBurger commented Jun 6, 2021

Uh oh!

codecov bot commented Jun 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JonnyBurger commented Aug 11, 2021

Uh oh!

JonnyBurger commented Aug 11, 2021

Uh oh!

Uh oh!

JonnyBurger commented Apr 15, 2021 •

edited

Loading

codecov bot commented Jun 6, 2021 •

edited

Loading