Skip to content

Blazor Server Large File Upload Support #33900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Components/Web.JS/dist/Release/blazor.server.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/Components/Web.JS/dist/Release/blazor.webview.js

Large diffs are not rendered by default.

21 changes: 9 additions & 12 deletions src/Components/Web.JS/src/InputFile.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

export const InputFile = {
init,
toImageFile,
Expand All @@ -14,6 +13,7 @@ interface BrowserFile {
contentType: string;
readPromise: Promise<ArrayBuffer> | undefined;
arrayBuffer: ArrayBuffer | undefined;
blob: Blob;
}

export interface InputElement extends HTMLInputElement {
Expand All @@ -33,7 +33,7 @@ function init(callbackWrapper: any, elem: InputElement): void {
// Reduce to purely serializable data, plus an index by ID.
elem._blazorFilesById = {};

const fileList = Array.prototype.map.call(elem.files, function(file): BrowserFile {
const fileList = Array.prototype.map.call(elem.files, function(file: File): BrowserFile {
const result = {
id: ++elem._blazorInputFileNextFileId,
lastModified: new Date(file.lastModified).toISOString(),
Expand All @@ -42,13 +42,11 @@ function init(callbackWrapper: any, elem: InputElement): void {
contentType: file.type,
readPromise: undefined,
arrayBuffer: undefined,
blob: file,
};

elem._blazorFilesById[result.id] = result;

// Attach the blob data itself as a non-enumerable property so it doesn't appear in the JSON.
Object.defineProperty(result, 'blob', { value: file });

return result;
});

Expand Down Expand Up @@ -78,6 +76,7 @@ async function toImageFile(elem: InputElement, fileId: number, format: string, m
canvas.getContext('2d')?.drawImage(loadedImage, 0, 0, canvas.width, canvas.height);
canvas.toBlob(resolve, format);
});

const result: BrowserFile = {
id: ++elem._blazorInputFileNextFileId,
lastModified: originalFile.lastModified,
Expand All @@ -86,13 +85,11 @@ async function toImageFile(elem: InputElement, fileId: number, format: string, m
contentType: format,
readPromise: undefined,
arrayBuffer: undefined,
blob: resizedImageBlob ? resizedImageBlob : originalFile.blob,
};

elem._blazorFilesById[result.id] = result;

// Attach the blob data itself as a non-enumerable property so it doesn't appear in the JSON.
Object.defineProperty(result, 'blob', { value: resizedImageBlob });

return result;
}

Expand All @@ -101,9 +98,9 @@ async function ensureArrayBufferReadyForSharedMemoryInterop(elem: InputElement,
getFileById(elem, fileId).arrayBuffer = arrayBuffer;
}

async function readFileData(elem: InputElement, fileId: number): Promise<Uint8Array> {
const arrayBuffer = await getArrayBufferFromFileAsync(elem, fileId);
return new Uint8Array(arrayBuffer);
async function readFileData(elem: InputElement, fileId: number): Promise<Blob> {
const file = getFileById(elem, fileId);
return file.blob;
}

export function getFileById(elem: InputElement, fileId: number): BrowserFile {
Expand All @@ -129,7 +126,7 @@ function getArrayBufferFromFileAsync(elem: InputElement, fileId: number): Promis
reader.onerror = function(err): void {
reject(err);
};
reader.readAsArrayBuffer(file['blob']);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't allocate a buffer larger than ~2GB so this'll fail for large files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this back in for now so this PR isn't blocked on WASM streaming interop. Will remove it in that PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this back in for now so this PR isn't blocked on WASM streaming interop. Will remove it in that PR.

Are you saying that, as of this PR, we still won't support >2GB files but it will nearly support it and then when you do the WASM piece it will support it on Server too?

If so that's totally fine - just want to check I understand.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of this PR we'll support >2GB in server and up to the current ~2GB limit in WASM. With the new WASM PR, I have >2GB working in WASM as well (using this Blob.slice approach).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TanayParikh where can I find this "the new WASM PR" and information weather Blazor WASAM 6.0 supports >2GB upload or not?

I can't find any documentation or examples on uploading >2GB file in WASM to an API (preferably with progressive information for UI display)? The docs on Blazor file upload doesn´t event mention the <2GB limit

Its totally unclear if this works now in .net 6.0 , just like I think you are pointing out here.

The internet is littered with questions like this and this and no answers.

Hope you can point me in the right direction.

Copy link
Contributor Author

@TanayParikh TanayParikh Dec 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where can I find this "the new WASM PR"

#33986

weather Blazor WASAM 6.0 supports >2GB upload or not?

It does.

I can't find any documentation or examples on uploading >2GB file in WASM to an API (preferably with progressive information for UI display)?

The process is the same as you would take to upload files < 2GB. We recently added docs for showing progress for uploads in Blazor server (here), you should be able to use that as guidance for implementing something similar in Blazor WASM.

The docs on Blazor file upload doesn´t event mention the <2GB limit

The 2GB limit in .NET 5 and before was un-intended and un-documented behavior. This has been resolved in .NET 6. I've created an issue to add a notice for this limitation in .NET 5 to the docs. dotnet/AspNetCore.Docs#24381

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks for the update. This helped me allot deciding on my next steps

reader.readAsArrayBuffer(file.blob);
});
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { HubConnection } from '@microsoft/signalr';

export function sendJSDataStream(connection: HubConnection, data: ArrayBufferView, streamId: string, chunkSize: number) {
export function sendJSDataStream(connection: HubConnection, data: ArrayBufferView | Blob, streamId: string, chunkSize: number) {
// Run the rest in the background, without delaying the completion of the call to sendJSDataStream
// otherwise we'll deadlock (.NET can't begin reading until this completes, but it won't complete
// because nobody's reading the pipe)
Expand All @@ -9,12 +9,13 @@ export function sendJSDataStream(connection: HubConnection, data: ArrayBufferVie
let numChunksUntilNextAck = 5;
let lastAckTime = new Date().valueOf();
try {
const byteLength = data instanceof Blob ? data.size : data.byteLength;
let position = 0;
let chunkId = 0;

while (position < data.byteLength) {
const nextChunkSize = Math.min(chunkSize, data.byteLength - position);
const nextChunkData = new Uint8Array(data.buffer, data.byteOffset + position, nextChunkSize);
while (position < byteLength) {
const nextChunkSize = Math.min(chunkSize, byteLength - position);
const nextChunkData = await getNextChunk(data, position, nextChunkSize);

numChunksUntilNextAck--;
if (numChunksUntilNextAck > 1) {
Expand Down Expand Up @@ -49,3 +50,23 @@ export function sendJSDataStream(connection: HubConnection, data: ArrayBufferVie
}
}, 0);
};

async function getNextChunk(data: ArrayBufferView | Blob, position: number, nextChunkSize: number): Promise<Uint8Array> {
if (data instanceof Blob) {
return await getChunkFromBlob(data, position, nextChunkSize);
} else {
return getChunkFromArrayBufferView(data, position, nextChunkSize);
}
}

async function getChunkFromBlob(data: Blob, position: number, nextChunkSize: number): Promise<Uint8Array> {
const chunkBlob = data.slice(position, position + nextChunkSize);
const arrayBuffer = await chunkBlob.arrayBuffer();
const nextChunkData = new Uint8Array(arrayBuffer);
return nextChunkData;
}

function getChunkFromArrayBufferView(data: ArrayBufferView, position: number, nextChunkSize: number) {
const nextChunkData = new Uint8Array(data.buffer, data.byteOffset + position, nextChunkSize);
return nextChunkData;
}
9 changes: 2 additions & 7 deletions src/Components/Web/src/Forms/InputFile/BrowserFileStream.cs
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,8 @@ public override Task<int> ReadAsync(byte[] buffer, int offset, int count, Cancel

public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
{
int maxBytesToRead = (int)(Length - Position);

if (maxBytesToRead > buffer.Length)
{
maxBytesToRead = buffer.Length;
}

var bytesAvailableToRead = Length - Position;
var maxBytesToRead = (int)Math.Min(bytesAvailableToRead, buffer.Length);
if (maxBytesToRead <= 0)
{
return 0;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ export module DotNet {
class JSObject {
_cachedFunctions: Map<string, Function>;

constructor(private _jsObject: any)
{
constructor(private _jsObject: any) {
this._cachedFunctions = new Map<string, Function>();
}

Expand Down Expand Up @@ -142,28 +141,40 @@ export module DotNet {
/**
* Creates a JavaScript data reference that can be passed to .NET via interop calls.
*
* @param arrayBufferView The ArrayBufferView used to create the JavaScript stream reference.
* @param streamReference The ArrayBufferView or Blob used to create the JavaScript stream reference.
* @returns The JavaScript data reference (this will be the same instance as the given object).
* @throws Error if the given value is not an Object or doesn't have a valid byteLength.
*/
export function createJSStreamReference(arrayBufferView: ArrayBufferView | any): any {
// Check if this is an ArrayBufferView, and if it has a valid byteLength for transfer
// using a JSStreamReference.
if (!(arrayBufferView.buffer instanceof ArrayBuffer)) {
throw new Error(`Cannot create a JSStreamReference from the value '${arrayBufferView}' as it is not have a 'buffer' property of type 'ArrayBuffer'.`);
} else if (arrayBufferView.byteLength === undefined) {
throw new Error(`Cannot create a JSStreamReference from the value '${arrayBufferView}' as it doesn't have a byteLength.`);
export function createJSStreamReference(streamReference: ArrayBuffer | ArrayBufferView | Blob | any): any {
let length = -1;

// If we're given a raw Array Buffer, we interpret it as a `Uint8Array` as
// ArrayBuffers' aren't directly readable.
if (streamReference instanceof ArrayBuffer) {
streamReference = new Uint8Array(streamReference);
}

if (streamReference instanceof Blob) {
length = streamReference.size;
} else if (streamReference.buffer instanceof ArrayBuffer) {
if (streamReference.byteLength === undefined) {
throw new Error(`Cannot create a JSStreamReference from the value '${streamReference}' as it doesn't have a byteLength.`);
}

length = streamReference.byteLength;
} else {
throw new Error('Supplied value is not a typed array or blob.');
}

const result: any = {
[jsStreamReferenceLengthKey]: arrayBufferView.byteLength,
[jsStreamReferenceLengthKey]: length,
}

try {
const jsObjectReference = createJSObjectReference(arrayBufferView);
const jsObjectReference = createJSObjectReference(streamReference);
result[jsObjectIdKey] = jsObjectReference[jsObjectIdKey];
} catch {
throw new Error(`Cannot create a JSStreamReference from the value '${arrayBufferView}'.`);
throw new Error(`Cannot create a JSStreamReference from the value '${streamReference}'.`);
}

return result;
Expand Down