Skip to content

Use interned Header Names for known headers not in the pre-allocated block #31311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
6 commits merged into from
Mar 30, 2021
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions src/Http/Headers/src/HeaderNames.cs
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ public static class HeaderNames
public static readonly string Authorization = "Authorization";

/// <summary>Gets the <c>baggage</c> HTTP header name.</summary>
public static readonly string Baggage = "baggage";
public static readonly string Baggage = "Baggage";

/// <summary>Gets the <c>Cache-Control</c> HTTP header name.</summary>
public static readonly string CacheControl = "Cache-Control";
Expand Down Expand Up @@ -166,6 +166,9 @@ public static class HeaderNames
/// <summary>Gets the <c>Last-Modified</c> HTTP header name.</summary>
public static readonly string LastModified = "Last-Modified";

/// <summary>Gets the <c>Link</c> HTTP header name.</summary>
public static readonly string Link = "Link";

/// <summary>Gets the <c>Location</c> HTTP header name.</summary>
public static readonly string Location = "Location";

Expand Down Expand Up @@ -245,10 +248,10 @@ public static class HeaderNames
public static readonly string Translate = "Translate";

/// <summary>Gets the <c>traceparent</c> HTTP header name.</summary>
public static readonly string TraceParent = "traceparent";
public static readonly string TraceParent = "TraceParent";

/// <summary>Gets the <c>tracestate</c> HTTP header name.</summary>
public static readonly string TraceState = "tracestate";
public static readonly string TraceState = "TraceState";

/// <summary>Gets the <c>Upgrade</c> HTTP header name.</summary>
public static readonly string Upgrade = "Upgrade";
Expand All @@ -274,10 +277,16 @@ public static class HeaderNames
/// <summary>Gets the <c>WWW-Authenticate</c> HTTP header name.</summary>
public static readonly string WWWAuthenticate = "WWW-Authenticate";

/// <summary>Gets the <c>X-Content-Type-Options</c> HTTP header name.</summary>
public static readonly string XContentTypeOptions = "X-Content-Type-Options";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you adding Link, X-Content-Type-Options and X-XSS-Protection here? We don't intend this list to be exhaustive, we primarily add entries that are used directly by the framework. It's not clear why these were selected for special treatment in Kestrel.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you adding Link, X-Content-Type-Options and X-XSS-Protection here?

www.webpagetest.org is one of the commonly used performance testing sites (along with Lighthouse based things for core web vitals e.g. web.dev/measure); and it will call you out; rightly or wrongly, for not having X-Content-Type-Options and X-XSS-Protection set; amongst others:

Funnily X-Content-Type-Options is the only one dotnet.microsoft.com does set:

image

image

vs

image

image

Just looks bad if you are explicitly called out for not setting them; but the Framework doesn't support them as a pre-defined headername when it has 90 other ones...

The Link header rfc5988; is a good one for using for preloading of fonts prior to the css being parsed and html starting to render to increase webpage performance and reduce flash of unstyled content (FOUC) or flash of invisible text (FOIT) if you mark the text as unrenderable until the font is downloaded.

e.g. Adding this header on the landing HTML page

link: <https://assets.ageofascent.net/fonts/teko-light-latin.woff2>; rel=preload; as=font; crossorigin=anonymous,<https://assets.ageofascent.net/fonts/pt_sans-web-regular.woff>; rel=preload; as=font; crossorigin=anonymous

Chrome/Edge will then immediately start downloading the fonts; without yet having encountered the fonts in the parsing of the HTML (triggered by evaluating its css)

image

The browsers will also tell you off if you then don't use the preloaded font within the first few seconds of page load:

image

Anyway Link is one of the missed headers I use; so fringe benefit in adding for me with this change 😉

Its not adding it to the pre-reserved Kestrel headers; just to the HeaderNames constants; as otherwise they look fugly in use:

image

Though since you made me look; seems should add "Referrer-Policy" also 🤔


/// <summary>Gets the <c>X-Frame-Options</c> HTTP header name.</summary>
public static readonly string XFrameOptions = "X-Frame-Options";

/// <summary>Gets the <c>X-Requested-With</c> HTTP header name.</summary>
public static readonly string XRequestedWith = "X-Requested-With";

/// <summary>Gets the <c>X-XSS-Protection</c> HTTP header name.</summary>
public static readonly string XXssProtection = "X-XSS-Protection";
}
}
3 changes: 3 additions & 0 deletions src/Http/Headers/src/PublicAPI.Unshipped.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@
Microsoft.Net.Http.Headers.MediaTypeHeaderValue.MatchesMediaType(Microsoft.Extensions.Primitives.StringSegment otherMediaType) -> bool
Microsoft.Net.Http.Headers.RangeConditionHeaderValue.RangeConditionHeaderValue(Microsoft.Net.Http.Headers.EntityTagHeaderValue! entityTag) -> void
static readonly Microsoft.Net.Http.Headers.HeaderNames.Baggage -> string!
static readonly Microsoft.Net.Http.Headers.HeaderNames.Link -> string!
static readonly Microsoft.Net.Http.Headers.HeaderNames.ProxyConnection -> string!
static readonly Microsoft.Net.Http.Headers.HeaderNames.XContentTypeOptions -> string!
static readonly Microsoft.Net.Http.Headers.HeaderNames.XXssProtection -> string!
7 changes: 3 additions & 4 deletions src/Middleware/WebSockets/src/Constants.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ namespace Microsoft.AspNetCore.WebSockets
internal static class Constants
{
public static class Headers
{
public const string UpgradeWebSocket = "websocket";
public const string ConnectionUpgrade = "Upgrade";
public const string SupportedVersion = "13";
{
public readonly static string UpgradeWebSocket = "websocket";
public readonly static string SupportedVersion = "13";
}
}
}
22 changes: 18 additions & 4 deletions src/Middleware/WebSockets/src/HandshakeHelpers.cs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ internal static class HandshakeHelpers
};

// Verify Method, Upgrade, Connection, version, key, etc..
public static bool CheckSupportedWebSocketRequest(string method, List<KeyValuePair<string, string>> headers)
public static bool CheckSupportedWebSocketRequest(string method, List<KeyValuePair<string, string>> interestingHeaders, IHeaderDictionary requestHeaders)
{
bool validUpgrade = false, validConnection = false, validKey = false, validVersion = false;

Expand All @@ -43,11 +43,11 @@ public static bool CheckSupportedWebSocketRequest(string method, List<KeyValuePa
return false;
}

foreach (var pair in headers)
foreach (var pair in interestingHeaders)
{
if (string.Equals(HeaderNames.Connection, pair.Key, StringComparison.OrdinalIgnoreCase))
{
if (string.Equals(Constants.Headers.ConnectionUpgrade, pair.Value, StringComparison.OrdinalIgnoreCase))
if (string.Equals(HeaderNames.Upgrade, pair.Value, StringComparison.OrdinalIgnoreCase))
{
validConnection = true;
}
Expand All @@ -72,12 +72,26 @@ public static bool CheckSupportedWebSocketRequest(string method, List<KeyValuePa
}
}

// WebSockets are long lived; so if the header values are valid we switch them out for the interned versions.
if (validConnection && requestHeaders[HeaderNames.Connection].Count == 1)
{
requestHeaders[HeaderNames.Connection] = HeaderNames.Upgrade;
}
if (validUpgrade && requestHeaders[HeaderNames.Upgrade].Count == 1)
{
requestHeaders[HeaderNames.Upgrade] = Constants.Headers.UpgradeWebSocket;
}
if (validVersion && requestHeaders[HeaderNames.SecWebSocketVersion].Count == 1)
{
requestHeaders[HeaderNames.SecWebSocketVersion] = Constants.Headers.SupportedVersion;
}

return validConnection && validUpgrade && validVersion && validKey;
}

public static void GenerateResponseHeaders(string key, string? subProtocol, IHeaderDictionary headers)
{
headers[HeaderNames.Connection] = Constants.Headers.ConnectionUpgrade;
headers[HeaderNames.Connection] = HeaderNames.Upgrade;
headers[HeaderNames.Upgrade] = Constants.Headers.UpgradeWebSocket;
headers[HeaderNames.SecWebSocketAccept] = CreateResponseKey(key);
if (!string.IsNullOrWhiteSpace(subProtocol))
Expand Down
11 changes: 6 additions & 5 deletions src/Middleware/WebSockets/src/WebSocketMiddleware.cs
Original file line number Diff line number Diff line change
Expand Up @@ -118,15 +118,16 @@ public bool IsWebSocketRequest
}
else
{
var headers = new List<KeyValuePair<string, string>>();
foreach (string headerName in HandshakeHelpers.NeededHeaders)
var requestHeaders = _context.Request.Headers;
var interestingHeaders = new List<KeyValuePair<string, string>>();
foreach (var headerName in HandshakeHelpers.NeededHeaders)
{
foreach (var value in _context.Request.Headers.GetCommaSeparatedValues(headerName))
foreach (var value in requestHeaders.GetCommaSeparatedValues(headerName))
{
headers.Add(new KeyValuePair<string, string>(headerName, value));
interestingHeaders.Add(new KeyValuePair<string, string>(headerName, value));
}
}
_isWebSocketRequest = HandshakeHelpers.CheckSupportedWebSocketRequest(_context.Request.Method, headers);
_isWebSocketRequest = HandshakeHelpers.CheckSupportedWebSocketRequest(_context.Request.Method, interestingHeaders, requestHeaders);
}
}
return _isWebSocketRequest.Value;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ public static MessageBody For(

if (headers.HasConnection)
{
var connectionOptions = HttpHeaders.ParseConnection(headers.HeaderConnection);
var connectionOptions = HttpHeaders.ParseConnection(headers);

upgrade = (connectionOptions & ConnectionOptions.Upgrade) != 0;
keepAlive = keepAlive || (connectionOptions & ConnectionOptions.KeepAlive) != 0;
Expand Down
104 changes: 102 additions & 2 deletions src/Servers/Kestrel/Core/src/Internal/Http/HttpHeaders.Generated.cs
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,106 @@ internal enum KnownHeaderType
WWWAuthenticate,
}

internal partial class HttpHeaders
{
protected readonly static HashSet<string> s_internedHeaderNames = new HashSet<string>(93, StringComparer.OrdinalIgnoreCase)
{
HeaderNames.Accept,
HeaderNames.AcceptCharset,
HeaderNames.AcceptEncoding,
HeaderNames.AcceptLanguage,
HeaderNames.AcceptRanges,
HeaderNames.AccessControlAllowCredentials,
HeaderNames.AccessControlAllowHeaders,
HeaderNames.AccessControlAllowMethods,
HeaderNames.AccessControlAllowOrigin,
HeaderNames.AccessControlExposeHeaders,
HeaderNames.AccessControlMaxAge,
HeaderNames.AccessControlRequestHeaders,
HeaderNames.AccessControlRequestMethod,
HeaderNames.Age,
HeaderNames.Allow,
HeaderNames.AltSvc,
HeaderNames.Authority,
HeaderNames.Authorization,
HeaderNames.Baggage,
HeaderNames.CacheControl,
HeaderNames.Connection,
HeaderNames.ContentDisposition,
HeaderNames.ContentEncoding,
HeaderNames.ContentLanguage,
HeaderNames.ContentLength,
HeaderNames.ContentLocation,
HeaderNames.ContentMD5,
HeaderNames.ContentRange,
HeaderNames.ContentSecurityPolicy,
HeaderNames.ContentSecurityPolicyReportOnly,
HeaderNames.ContentType,
HeaderNames.CorrelationContext,
HeaderNames.Cookie,
HeaderNames.Date,
HeaderNames.DNT,
HeaderNames.ETag,
HeaderNames.Expires,
HeaderNames.Expect,
HeaderNames.From,
HeaderNames.GrpcAcceptEncoding,
HeaderNames.GrpcEncoding,
HeaderNames.GrpcMessage,
HeaderNames.GrpcStatus,
HeaderNames.GrpcTimeout,
HeaderNames.Host,
HeaderNames.KeepAlive,
HeaderNames.IfMatch,
HeaderNames.IfModifiedSince,
HeaderNames.IfNoneMatch,
HeaderNames.IfRange,
HeaderNames.IfUnmodifiedSince,
HeaderNames.LastModified,
HeaderNames.Link,
HeaderNames.Location,
HeaderNames.MaxForwards,
HeaderNames.Method,
HeaderNames.Origin,
HeaderNames.Path,
HeaderNames.Pragma,
HeaderNames.ProxyAuthenticate,
HeaderNames.ProxyAuthorization,
HeaderNames.ProxyConnection,
HeaderNames.Range,
HeaderNames.Referer,
HeaderNames.RetryAfter,
HeaderNames.RequestId,
HeaderNames.Scheme,
HeaderNames.SecWebSocketAccept,
HeaderNames.SecWebSocketKey,
HeaderNames.SecWebSocketProtocol,
HeaderNames.SecWebSocketVersion,
HeaderNames.Server,
HeaderNames.SetCookie,
HeaderNames.Status,
HeaderNames.StrictTransportSecurity,
HeaderNames.TE,
HeaderNames.Trailer,
HeaderNames.TransferEncoding,
HeaderNames.Translate,
HeaderNames.TraceParent,
HeaderNames.TraceState,
HeaderNames.Upgrade,
HeaderNames.UpgradeInsecureRequests,
HeaderNames.UserAgent,
HeaderNames.Vary,
HeaderNames.Via,
HeaderNames.Warning,
HeaderNames.WebSocketSubProtocols,
HeaderNames.WWWAuthenticate,
HeaderNames.XContentTypeOptions,
HeaderNames.XFrameOptions,
HeaderNames.XRequestedWith,
HeaderNames.XXssProtection,
};
}

internal partial class HttpRequestHeaders
{
private HeaderReferences _headers;
Expand All @@ -125,7 +225,7 @@ public StringValues HeaderCacheControl
_headers._CacheControl = value;
}
}
public StringValues HeaderConnection
public override StringValues HeaderConnection
{
get
{
Expand Down Expand Up @@ -8129,7 +8229,7 @@ public StringValues HeaderCacheControl
_headers._CacheControl = value;
}
}
public StringValues HeaderConnection
public override StringValues HeaderConnection
{
get
{
Expand Down
61 changes: 57 additions & 4 deletions src/Servers/Kestrel/Core/src/Internal/Http/HttpHeaders.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,17 @@
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure;
using Microsoft.Extensions.Primitives;
using Microsoft.Net.Http.Headers;

namespace Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http
{
internal abstract class HttpHeaders : IHeaderDictionary
internal abstract partial class HttpHeaders : IHeaderDictionary
{
protected long _bits = 0;
protected long? _contentLength;
protected bool _isReadOnly;
protected Dictionary<string, StringValues>? MaybeUnknown;
protected Dictionary<string, StringValues> Unknown => MaybeUnknown ?? (MaybeUnknown = new Dictionary<string, StringValues>(StringComparer.OrdinalIgnoreCase));
protected Dictionary<string, StringValues> Unknown => MaybeUnknown ??= new Dictionary<string, StringValues>(StringComparer.OrdinalIgnoreCase);

public long? ContentLength
{
Expand All @@ -40,6 +41,8 @@ public long? ContentLength
}
}

public abstract StringValues HeaderConnection { get; set; }

StringValues IHeaderDictionary.this[string key]
{
get
Expand Down Expand Up @@ -126,6 +129,18 @@ public void Reset()
ClearFast();
}

protected static string GetInternedHeaderName(string name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method looks like it will be used to set unknown headers.

Aren't most of the headers in this collection known? They will never be used because generated code will handle setting them, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are 93 header names in HeaderNames; but only 20 in commonHeaders shared by HttpRequestHeaders and HttpReponseHeaders; could drop those 20; or have a different HashSet<string> for request and response but it doesn't seem worth it?

However yes, if the header is in the set for HttpRequestHeaders or HttpReponseHeaders and comes in on that direction it won't use the HashSet<string>, it only uses it for header names that aren't in that set rather than making a freshly constructed string long lived.

Any headers outside the set in HeaderNames will remain as the newly constructed string and won't be exchanged (i.e. the HashSet<string> is not updated while the app is running)

Copy link
Member

@JamesNK JamesNK Mar 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok!

This two phases of hash lookups (intern hashset then unknown dictionary) makes me wonder whether it makes sense to write a custom collection in the future.

As well as optimizing interning known keys, there is probably an opportunity to optimize memory usage. Most requests have few unknown headers, so O(n) lookups wouldn't matter. e.g. an adaptive collection that can switch between KeyValuePair<string, StringValues>[] and Dictionary<string, StringValues> depending upon how many headers have been added.

Copy link
Member Author

@benaadams benaadams Mar 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any headers outside the set in HeaderNames will remain as the newly constructed string and won't be exchanged

Is a question on whether to extend the values in HeaderNames and it becomes more valuable to do so with this change, rather than just being coinvent for the developer; e.g. missed standardise headers and defacto standard headers: Prefer, Preference-Applied, X-Request-ID, X-Correlation-ID, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Proto, Refresh, X-Powered-By, Timing-Allow-Origin

Though response set headers would likely be less important than request set headers; assuming they are being set by a single assembly so already an interned value (beyond developer convenience); though a pass-through name where Kestrel was acting as a proxy would still be valuable.

However that's probably a separate issue...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added issue for other headers #31374

{
// Some headers can be very long lived; for example those on a WebSocket connection
// so we exchange these for the preallocated strings predefined in HeaderNames
if (s_internedHeaderNames.TryGetValue(name, out var internedName))
{
return internedName;
}

return name;
}

[MethodImpl(MethodImplOptions.NoInlining)]
protected static StringValues AppendValue(StringValues existing, string append)
{
Expand Down Expand Up @@ -276,7 +291,12 @@ public static void ValidateHeaderNameCharacters(string headerCharacters)
}
}

public static ConnectionOptions ParseConnection(StringValues connection)
private readonly static string KeepAlive = "keep-alive";
private readonly static StringValues ConnectionValueKeepAlive = KeepAlive;
private readonly static StringValues ConnectionValueClose = "close";
private readonly static StringValues ConnectionValueUpgrade = HeaderNames.Upgrade;

public static ConnectionOptions ParseConnection(HttpHeaders headers)
{
// Keep-alive
const ulong lowerCaseKeep = 0x0000_0020_0020_0020; // Don't lowercase hyphen
Expand All @@ -289,9 +309,27 @@ public static ConnectionOptions ParseConnection(StringValues connection)
// Close
const ulong closeEnd = 0x0065_0073_006f_006c; // 4 chars "lose"

var connection = headers.HeaderConnection;
var connectionCount = connection.Count;
if (connectionCount == 0)
{
return ConnectionOptions.None;
}

var connectionOptions = ConnectionOptions.None;

var connectionCount = connection.Count;
if (connectionCount == 1)
{
// "keep-alive" is the only value that will be repeated over
// many requests on the same connection; on the first request
// we will have switched it for the readonly static value;
// so we can ptentially short-circuit parsing and use ReferenceEquals.
if (ReferenceEquals(connection.ToString(), KeepAlive))
{
return ConnectionOptions.KeepAlive;
}
}

for (var i = 0; i < connectionCount; i++)
{
var value = connection[i].AsSpan();
Expand Down Expand Up @@ -420,6 +458,21 @@ public static ConnectionOptions ParseConnection(StringValues connection)
}
}

// If Connection is a single value, switch it for the interned value
// in case the connection is long lived
if (connectionOptions == ConnectionOptions.Upgrade)
{
headers.HeaderConnection = ConnectionValueUpgrade;
}
else if (connectionOptions == ConnectionOptions.KeepAlive)
{
headers.HeaderConnection = ConnectionValueKeepAlive;
}
else if (connectionOptions == ConnectionOptions.Close)
{
headers.HeaderConnection = ConnectionValueClose;
}

return connectionOptions;
}

Expand Down
Loading