[ResultBuilders] `buildBlock(combining:into:)` for pairwise combination. #40799

rxwei · 2022-01-11T18:59:22Z

Allow a user-defined buildBlock(combining:into:) to combine subexpressions in a block pairwise top to bottom. The feature is being gated under frontend flag -enable-experimental-pairwise-build-block. To use buildBlock(combining:into:), the user also needs to provide a unary buildBlock(_:) as a base case. When buildBlock(combining:into:) is provided, a block will be transformed to the following:

{
    arg_0,
    arg_1,
    arg_2,
    ...,
    arg_n
}

let v0 = Builder.buildBlock(arg_0)
let v1 = Builder.buildBlock(combining: arg_1, into: v0)
...
return Builder.buildBlock(combining: arg_n, into: ...)

This will enable use cases in RegexBuilder in experimental declarative string processing, where we need to concatenate tuples and conditionally skip captureless regexes. For example:

let regex = Regex {
  "a" // Regex<Substring>
  OneOrMore("b").capture() // Regex<(Substring, Substring)>
  "c" // Regex<Substring>
  Optionally("d".capture()) // Regex<(Substring, Substring?)>
} // Regex<(Substring, Substring, Substring?)>
let result = "abc".firstMatch(of: regex)
// MatchResult<(Substring, Substring, Substring?)>

In this example, patterns "a" and "c" have no captures, so we need to skip their capture types in the resulting generic parameter type rather than record them as (). However with the existing result builder buildBlock() feature that builds a block wholesale from all subexpressions, we had to generate 2^arity overloads accounting for any occurrences of captureless regexes. There are also other complexities such as having to drop-first from the tuple to obtain the capture type. Though these features could in theory be supported via variadic generics, we feel that allowing result builders to pairwise combine subexpressions in a block is a much simpler and potentially more useful approach.

For details about regex captures, see Strongly Typed Regex Captures.

With buildBlock(combining:into:), the regex builders can be defined as the following, assuming we have variadic generics:

enum RegexBuilder {
  static func buildBlock() -> Regex<Substring>
  static func buildBlock<Match>(_ x: Regex<Match>) -> Regex<Match>
  static func buildBlock<ExistingWholeMatch, NewWholeMatch, ExistingCaptures..., NewCaptures...>(
    combining next: Regex<(NewWholeMatch, NewCaptures...)>,
    into combined: Regex<(ExistingWholeMatch,  ExistingCaptures...)>
  ) -> Regex<(Substring, ExistingCaptures..., NewCaptures...)>
  static func buildBlock<ExistingWholeMatch, NewWholeMatch, ExistingCaptures...>(
    combining next: Regex<(NewWholeMatch, [Void])>,
    into combined: Regex<(ExistingWholeMatch,  ExistingCaptures...)>
  ) -> Regex<(Substring, ExistingCaptures...)>
  static func buildBlock<ExistingWholeMatch, NewWholeMatch, ExistingCaptures...>(
    combining next: Regex<(NewWholeMatch, Void?)>,
    into combined: Regex<(ExistingWholeMatch,  ExistingCaptures...)>
  ) -> Regex<(Substring, ExistingCaptures...)>
}

Before we have variadic generics, we can define overloads of buildBlock(combining:into:) for up to a certain arity. These overloads will be much fewer than 2^arity.

rxwei · 2022-01-11T22:47:07Z

To reviewers: If you agree this is the right direction, I'll go ahead and add more tests and hopefully more diagnostics.

rxwei · 2022-01-12T00:53:09Z

@swift-ci Please Build Toolchain macOS Platform

rxwei · 2022-01-12T08:10:05Z

Toolchain: https://ci.swift.org/job/swift-PR-toolchain-osx/1279/artifact/branch-main/swift-PR-40799-1279-osx.tar.gz

DougGregor

This implementation looks really good. Approve from the PR-is-good-once-it-has-more-tests standpoint, but needs evolution.

DougGregor · 2022-01-12T17:12:10Z

lib/Sema/BuilderTransform.cpp

+      //         combining: arg_1, into: Builder.buildBlock(arg_0))...
+      call = buildCallIfWanted(
+          braceStmt->getStartLoc(), ctx.Id_buildBlock, {expressions.front()},
+          /*argLabels=*/{});


It might be cleaner to say that we call buildBlock() initially to get the first call, then the expressions (if there are any) are combined with buildBlock(_combining:into:). That way, we don't need the expressions.empty() check in the if statement.

If we called the nullary buildBlock() as a base case, the result builder would have to to support an empty block. Should the developer be allowed to define a DSL that doesn't support empty blocks?

DougGregor · 2022-01-12T17:13:07Z

This idea has been around for quite a while.... see https://forums.swift.org/t/function-builders/25167/311.

rxwei · 2022-01-12T20:25:40Z

This idea has been around for quite a while.... see https://forums.swift.org/t/function-builders/25167/311.

Cool. It now looks like the pairwise formulation is beyond a workaround for variadic generics.

xedin · 2022-01-13T18:34:37Z

error: the compiler is unable to type-check this expression in reasonable time; try breaking up the expression into distinct sub-expressions

This is something that I'm concerned about with this changes, but I haven't had a chance to look at the PR yet. We need to set it up in a way where the chain builds up incrementally and doesn't need to re-typecheck its previous components otherwise we'd end up with giant constraint system which solver wouldn't be able to handle...

rxwei · 2022-01-13T18:36:43Z

@xedin does breaking it up to multiple statements sound good to you?

let v0 = Builder.buildBlock(arg_0)
let v1 = Builder.buildBlock(_combining: arg_1, into: v0)
...
let vn = Builder.buildBlock(_combining: arg_n, into: ...)
return vn

xedin · 2022-01-13T18:39:18Z

@rxwei Yeah, that sounds good.

rxwei · 2022-01-13T18:46:13Z

@DougGregor Regarding your suggestion to call nullary buildBlock() first, if the DSL needs the first subexpression to infer the type, I don't think this will work if we break everything down to separate statements, unless I make the first statement be something like:

let v1 = Builder.buildBlock(_combining: arg_1, into: buildBlock())

Which approach is better? To me it seems like calling unary buildBlock as a base case (#40799 (comment)) makes it more flexible. The developer can always add a nullary buildBlock to support empty blocks.

rxwei · 2022-01-15T03:52:25Z

@swift-ci please build toolchain macOS platform

swift-ci · 2022-01-15T13:15:40Z

macOS Toolchain
Download Toolchain
Git Sha - 9c84d355758f87243211e3193dc3d79fa0749785

Install command
tar -zxf swift-PR-40799-1289-osx.tar.gz --directory ~/

…o:)`. This patch replaces `O(2^arity)` overloads of `buildBlock` with `O(arity^2)` overloads of `buildBlock(combining:into:)` with the language feature implemented in swiftlang/swift#40799. This improves library compilation time and fixes an issue where nested captures did not have a flattened type. This also allows us to switch back to native tuples for `Match`, which will be addressed in a follow-up patch.

rxwei · 2022-01-18T13:04:58Z

I ran into a strange SILGen crash when generating additional let bindings and thought it's easier to add one-way constraints in between subexpressions, so I did that instead. This made swiftlang/swift-experimental-string-processing#114 compile in totally reasonable time.

Allow a user-defined `buildBlock(combining:into:)` to combine subexpressions in a block pairwise top to bottom. To use `buildBlock(_combining:into:)`, the user also needs to provide a unary `buildBlock(_:)` as a base case. The feature is being gated under frontend flag `-enable-experimental-pairwise-build-block`. This will enable use cases in `RegexBuilder` in experimental declarative string processing, where we need to concatenate tuples and conditionally skip captureless regexes. For example: ```swift let regex = Regex { "a" // Regex<Substring> OneOrMore("b").capture() // Regex<(Substring, Substring)> "c" // Regex<Substring> Optionally("d".capture()) // Regex<(Substring, Substring?)> } // Regex<Tuple3<Substring, Substring, Substring?>> let result = "abc".firstMatch(of: regex) // MatchResult<(Substring, Substring, Substring?)> ``` In this example, patterns `"a"` and `"c"` have no captures, so we need to skip them. However with the existing result builder `buildBlock()` feature that builds a block wholesale from all subexpressions, we had to generate `2^arity` overloads accounting for any occurrences of captureless regexes. There are also other complexities such as having to drop-first from the tuple to obtain the capture type. Though these features could in theory be supported via variadic generics, we feel that allowing result builders to pairwise combine subexpressions in a block is a much simpler and potentially more useful approach. With `buildBlock(_combining:into:)`, the regex builders can be defined as the following, assuming we have variadic generics: ```swift enum RegexBuilder { static func buildBlock() -> Regex<Substring> static func buildBlock<Match>(_ x: Regex<Match>) -> Regex<Match> static func buildBlock< ExistingWholeMatch, NewWholeMatch, ExistingCaptures..., NewCaptures... >( _combining next: Regex<(NewWholeMatch, NewCaptures...)>, into combined: Regex<(ExistingWholeMatch, ExistingCaptures...)> ) -> Regex<Substring, ExistingCaptures..., NewCaptures...> } ``` Before we have variadic generics, we can define overloads of `buildBlock(_combining:into:)` for up to a certain arity. These overloads will be much fewer than `2^arity`.

rxwei · 2022-01-18T13:14:23Z

@swift-ci please test

swift-ci · 2022-01-18T13:17:24Z

Build failed
Swift Test Linux Platform
Git Sha - b6e679f

rxwei · 2022-01-18T19:11:54Z

@swift-ci please benchmark

swift-ci · 2022-01-18T20:16:10Z

Performance (x86_64): -O

Regression	OLD	NEW	DELTA	RATIO
ArrayAppendGenericStructs	1330	2480	+86.5%	0.54x (?)

Improvement	OLD	NEW	DELTA	RATIO
DictionaryBridgeToObjC_BulkAccess	169	143	-15.4%	1.18x (?)
DictionaryKeysContainsCocoa	31	27	-12.9%	1.15x (?)

Code size: -O

Performance (x86_64): -Osize

Regression	OLD	NEW	DELTA	RATIO
ArrayAppendGenericStructs	1340	2480	+85.1%	0.54x (?)

Improvement	OLD	NEW	DELTA	RATIO
FlattenListFlatMap	6658	4367	-34.4%	1.52x (?)

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

stephentyrone · 2022-01-20T00:05:13Z

@swift-ci please smoke test compiler performance

rxwei · 2022-01-20T00:06:24Z

Oh, I triggered the wrong benchmark.

swift-ci · 2022-01-20T01:27:39Z

!!! Couldn't read commit file !!!

rxwei · 2022-01-20T05:46:36Z

@swift-ci
!!! Couldn't trigger performance tests !!!

…o:)`. This patch replaces `O(2^arity)` overloads of `buildBlock` with `O(arity^2)` overloads of `buildBlock(combining:into:)` with the language feature implemented in swiftlang/swift#40799. This improves library compilation time and fixes an issue where nested captures did not have a flattened type. This also allows us to switch back to native tuples for `Match`, which will be addressed in a follow-up patch.

…o:)`. This patch replaces `O(2^arity)` overloads of `buildBlock` with `O(arity^2)` overloads of `buildBlock(combining:into:)` with the language feature implemented in swiftlang/swift#40799. This improves library compilation time, and fixes an issue where nested captures did not have a flattened type. Before this patch, fixing this would need `O(arity!)` overloads. This also allows us to switch back to native tuples for `Match`.

…o:)`. This patch replaces `O(2^arity)` overloads of `buildBlock` with `O(arity^2)` overloads of `buildBlock(combining:into:)` with the language feature implemented in swiftlang/swift#40799. This improves library compilation time, and fixes an issue where nested captures did not have a flattened type. Before this patch, fixing this would need `O(arity!)` overloads. This also allows us to switch back to native tuples for `Match`. Also, rename `renderAsPattern` to `renderAsBuilderDSL` to work around a CI stale file issue.

…o:)`. This patch replaces `O(2^arity)` overloads of `buildBlock` with `O(arity^2)` overloads of `buildBlock(combining:into:)` with the language feature implemented in swiftlang/swift#40799. This improves library compilation time and fixes an issue where nested captures did not have a flattened type. This also allows us to switch back to native tuples for `Match`, which will be addressed in a follow-up patch.

rxwei changed the title ~~[ResultBuilders] buildBlock(joining:_:) for pairwise combination.~~ [Draft] [ResultBuilders] buildBlock(joining:_:) for pairwise combination. Jan 11, 2022

rxwei force-pushed the pairwise-buildblock branch from 35afa20 to 6d87816 Compare January 11, 2022 22:39

rxwei changed the title ~~[Draft] [ResultBuilders] buildBlock(joining:_:) for pairwise combination.~~ [Draft] [ResultBuilders] buildBlock(_combining:into:) for pairwise combination. Jan 11, 2022

rxwei force-pushed the pairwise-buildblock branch from 6d87816 to 8a329e4 Compare January 11, 2022 22:41

rxwei requested review from DougGregor and hborla January 11, 2022 22:45

DougGregor approved these changes Jan 12, 2022

View reviewed changes

rxwei force-pushed the pairwise-buildblock branch 5 times, most recently from 9ab4201 to 9c84d35 Compare January 15, 2022 03:52

rxwei mentioned this pull request Jan 18, 2022

Reduce buildBlock overloads with pairwise buildBlock(combining:into:) swiftlang/swift-experimental-string-processing#114

Merged

rxwei force-pushed the pairwise-buildblock branch from 9c84d35 to c821717 Compare January 18, 2022 12:59

rxwei changed the title ~~[Draft] [ResultBuilders] buildBlock(_combining:into:) for pairwise combination.~~ [ResultBuilders] buildBlock(_combining:into:) for pairwise combination. Jan 18, 2022

rxwei marked this pull request as ready for review January 18, 2022 13:01

rxwei force-pushed the pairwise-buildblock branch from c821717 to b6e679f Compare January 18, 2022 13:14

swiftlang deleted a comment from swift-ci Jan 18, 2022

rxwei changed the title ~~[ResultBuilders] buildBlock(_combining:into:) for pairwise combination.~~ [ResultBuilders] buildBlock(combining:into:) for pairwise combination. Jan 18, 2022

rxwei merged commit 59ad670 into swiftlang:main Jan 20, 2022

rxwei deleted the pairwise-buildblock branch January 20, 2022 01:11

rxwei restored the pairwise-buildblock branch January 20, 2022 01:11

[ResultBuilders] buildBlock(combining:into:) for pairwise combination. #40799

[ResultBuilders] buildBlock(combining:into:) for pairwise combination. #40799

Uh oh!

Conversation

rxwei commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rxwei commented Jan 11, 2022

Uh oh!

rxwei commented Jan 12, 2022

Uh oh!

rxwei commented Jan 12, 2022

Uh oh!

DougGregor left a comment

Choose a reason for hiding this comment

Uh oh!

DougGregor Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

rxwei Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

DougGregor commented Jan 12, 2022

Uh oh!

rxwei commented Jan 12, 2022

Uh oh!

xedin commented Jan 13, 2022

Uh oh!

rxwei commented Jan 13, 2022

Uh oh!

xedin commented Jan 13, 2022

Uh oh!

rxwei commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rxwei commented Jan 15, 2022

Uh oh!

swift-ci commented Jan 15, 2022

Uh oh!

rxwei commented Jan 18, 2022

Uh oh!

rxwei commented Jan 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swift-ci commented Jan 18, 2022

Uh oh!

rxwei commented Jan 18, 2022

Uh oh!

swift-ci commented Jan 18, 2022

Performance (x86_64): -O

Code size: -O

Performance (x86_64): -Osize

Code size: -Osize

Performance (x86_64): -Onone

Code size: -swiftlibs

Uh oh!

stephentyrone commented Jan 20, 2022

Uh oh!

rxwei commented Jan 20, 2022

Uh oh!

swift-ci commented Jan 20, 2022

Uh oh!

rxwei commented Jan 20, 2022

Uh oh!

Uh oh!

[ResultBuilders] `buildBlock(combining:into:)` for pairwise combination. #40799

[ResultBuilders] `buildBlock(combining:into:)` for pairwise combination. #40799

rxwei commented Jan 11, 2022 •

edited

Loading

rxwei commented Jan 13, 2022 •

edited

Loading

rxwei commented Jan 18, 2022 •

edited

Loading