Improve generation and use of ref/ projects

[dougbu]

This PR is a fairly large cleanup of our 'release/3.0' work to at least get implementation assemblies built against ref/ assemblies. It has the following aims:

• remove workarounds to the extent possible
• where the above isn't possible, centralize workarounds (for simplicity and to remove redundancy)
• avoid overloading the compiler with an extreme list of referenced assemblies
• eliminate as much complexity as possible in the relevant infrastructure

Stretch goals include:

• limit bloat of the ref/ assemblies
• improve ref/ project generation bootstrapping (speedup, avoid unexpected errors)

[dougbu]

This PR is incomplete. Please see Build.log (containing ~560 errors) and Build.unique.log (containing the ~190 unique errors).

In addition, the src\Identity\Extensions.Stores\ref\Microsoft.Extensions.Identity.Stores.Manual.cs file is empty. Just waiting to find what types we're using.

[dougbu]

Steps to handle the internal members that test, sample and perf code is complaining about:

1. find the relevant *.cs file on disk
2. navigate to the containing src/ folder
3. dotnet msbuild /t:GenerateReferenceSource /p:AdditionalGenApiCmdOptions="--all"
4. cp ..\ref\*.netcoreapp3.0.cs ..\ref\save.cs
5. cp ..\ref\*.netcoreapp3.0.cs ..\ref\{project}.Manual.cs # if file doesn't exist yet
6. dotnet msbuild /t:GenerateReferenceSource # easy to forget! (don't forget)
7. if the *.Manual.cs file didn't exist, pare it down to the missing members
   • otherwise, find the missing members and copy them into the existing *.Manual.cs file
8. remove almost all private members and associated attributes
   • replace __dummy* fields in structs with those from implementation type
     • GenAPI sometimes doesn't generate fields correctly, even if public
     • copying the actual fields also ensures sizeof(...) works properly and doesn't hurt anything
     • leave __dummy and __dummyPrimitive fields alone only if implementation struct has no fields
   • leave private constructors calling a base constructor
   • remove [AsyncStateMachine] attributes; they almost always refer to private classes
9. dotnet msbuild ..\test\ # to see what is still missing in the ref/ project you're changing

Suggest splitting up by ref/ project area to avoid both working on ProjectA because it contains missing members at opposite ends of the alphabet.

FYI I grabbed the warnings-only log using "/flp:v=quiet;logfile=.\artifacts\log\Debug\Build.log" on the build command line.

[dougbu]

Most urgent thing is to test building (and executing) variants of e.g. src\SignalR\samples\ClientSample outside the AspNetCore repo i.e. with the 3.0.1 targeting pack / SDK and all @(Reference) items turned into @(PackageReference) items. If there be dragons, may need something drastic like building more of Extensions and all of EF Core against ref/ assemblies. (Yes, the second would be very strange given EF Core packages target .NET Standard. Worst case, the projects would need to be multi-targeted.) I'm of course hoping no problems arise and we can paper over the issues with a few additional direct references on Extensions packages.

I suggest changing Extensions very slightly to build Microsoft.Extensions.Logging.Testing and maybe one or two other internal packages against ref/ assemblies.

[dougbu]

First 4 commits have already been merged -- in #17245. Visible because I changed the base branch to 'release/3.0' and GitHub hasn't noticed those changes are already there.

[dougbu]

Changed my mind and rebased the branch. See you all in a few days.

[dougbu]

Rebased to pick up a few more fixes e.g. @wtgodbe's cafb50c and 89aae45

[dougbu]

Bit more about handling private members: The main reasons __dummy, __dummyPrimitive and similar private fields exist in what GenAPI generates are covered in https://blog.paranoidcoding.com/2016/02/15/are-private-members-api-surface.html

So, the private member handling I described above is correct but involves more copying than absolutely required. Because sizeof(...) is usually evaluated at runtime, copying all of a struct's private fields isn't always mandatory. But, it is mandatory to check the real (implementation) type, to copy public and internal fields, and to copy everything when [StructLayout] is specified on the real type i.e. might as well copy the fields over.

Note: Any [StructLayout] on the real type means the fields must be copied over w/ their offsets or (if LayoutKind.Sequential) in the correct order.

[wtgodbe]

The only remaining non-CS1705 is:

F:\workspace_work\1\s\src\Components\Web.JS\Microsoft.AspNetCore.Components.Web.JS.npmproj : error MSB4057: The target "GetTargetPath" does not exist in the project.

Plus a couple NU5128 warnings:

F:\workspace_work\1\s.dotnet\sdk\3.0.101\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below: [F:\workspace_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\Symbols\Microsoft.AspNetCore.ANCMSymbols.csproj]
F:\workspace_work\1\s.dotnet\sdk\3.0.101\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: - Add lib or ref assemblies for the netcoreapp3.0 target framework [F:\workspace_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\Symbols\Microsoft.AspNetCore.ANCMSymbols.csproj]
F:\workspace_work\1\s.dotnet\sdk\3.0.101\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below: [F:\workspace_work\1\s\src\Analyzers\shared\FeatureDetection\Microsoft.AspNetCore.Analyzers.FeatureDetection.Sources.csproj]
F:\workspace_work\1\s.dotnet\sdk\3.0.101\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): warning NU5128: - Add lib or ref assemblies for the netstandard1.0 target framework [F:\workspace_work\1\s\src\Analyzers\shared\FeatureDetection\Microsoft.AspNetCore.Analyzers.FeatureDetection.Sources.csproj]

Source build is fixed now!