Re-enable signing validation

[dougbu]

• Re-enable signing validation #13864
• update exclusions to get them working

Summary

This PR re-enables signing validation for official builds of this repo. We unfortunately had to disable such validation in the 3.0 GA release due to problems in the validation tool. Those problems have been resolved.

Change is larger than changing one parameter to the post-build stages because

1. Need the latest Arcade SDK from the '.NET 3 Tools' channel to get the fixed validation tool
2. The new Arcade SDK supports embedded package icons and that support conflicts with what we did manually for 3.0 GA

Customer Impact

Increased confidence in the quality of our release and increased ability to be sure packages come from Microsoft

Regression?

No

Testing

See https://dev.azure.com/dnceng/internal/_build/results?buildId=418214&view=results trial build

Risk

Low. Signing validation appears to be quite deterministic and has passed in the build above.

[dougbu]

We'll see how https://dev.azure.com/dnceng/internal/_build/results?buildId=349950 goes…

[dougbu]

@joeloff that internal build didn't validate correct, same 98 unsigned files reported. Please have a look at this PR and https://dev.azure.com/dnceng/internal/_build/results?buildId=349950&view=logs&s=3df7d716-4c9c-5c26-9f45-11f62216640d&j=b11b921d-8982-5bb3-754b-b114d42fd804 and let me know what's going on. Much appreciated.

[joeloff]

According to the logs the 98 files are excluded. Arcade is flagging it as an error

All files are signed.
Total Time: 00:01:53.6841038
Total Files: 4824, Signed: 2764, Unsigned: 0, Skipped: 1962, Excluded: 98, Skipped & Excluded: 0
D:\a\1\s.packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19458.2\tools\SdkTasks\SigningValidation.proj(56,5): error : Signing validation failed. Check signcheck.errors.log for more information.

Wonder if it might be returning a non-zero result.

[joeloff]

@dougbu, I think I know what the problem is. The external cabinet files for the MSIS are not in the nupkg:

Could not find file 'C:\Users\VssAdministrator\AppData\Local\Temp\SignCheck\x4o2pvoj.sh1\sfx_x64.cab'.
Could not find file 'C:\Users\VssAdministrator\AppData\Local\Temp\SignCheck\mtaifw2m.rk2\sfx_x86.cab'.

[dougbu]

@joeloff we haven't had to include those files in the .nupkg before. I suspect the verification infrastructure in Arcade is different from what we were doing before.

In any case, I seem to recall we did something to at least archive those files not that long ago. Which .nupkg should they be in? That is, what's the fix?

[joeloff]

The nupkgs you're pushing to VS contain the MSI and CAB file. However, the folder to where the SignCheck tool is being pointed only has the MSI and not the CAB file.

The sign validation binlog will have the information, but I haven't been able to locate that file yet.

[joeloff]

According to the build step: D:\a\1\s\artifacts\log\Debug\SigningValidation.binlog

I thought it might be in the Windows_logs artifact, but don't see it there.

[dougbu]

Unfortunately, the post-build stages don't seem to create any new artifacts. @riarenas and @JohnTortugo is attempting to validate signing locally after downloading the artifacts the only workaround?

[dougbu]

And, @joeloff I thought you were able to reproduce problems locally? If not, can you try the validation using one of the offending .nupkg files from that build?

[JohnTortugo]

You should be able to repro this locally. The artifacts that SignCheck run upon should be the same ones that you download from the Artifacts link in the UI.

[joeloff]

The issue I found is with the exclusion itself. If you don't specify any wildcards, the exclusion is used literally.

If you look at the blob artifacts, you'll see that the MSI without the cab is published. This is likely what's being used. If the CAB is present next to the MSI you don't see the error.

The exclusions work as I pointed out earlier. The problem is that whichever folder or file pattern is assed to sign check points to a location that has an MSI and not the external CAB. That will cause the failures because the MSI lists all the files inside it, but their not in the MSI and the CAB they're supposed to be in is missing from the location.

I'll try and give you a call sometime

[joeloff]

So I found the issue. The problem is with how archives are processed. When the MSI is encountered prior to the external cab, extracting the MSI content would file. We first need to extract the archive and then verify its contents. The current implementation extract and immediately verifies each file. We also need to account for duplicate cabs in different folders inside zips/nupkgs/vsix files.

@dougbu, touching Arcade would propagate through all the 3.0 branches I think. Is there a way to keep the step active, but ignore the error and continue? That way we can still visually inspect the results.

[dougbu]

No longer blocked. Signing worked in https://dev.azure.com/dnceng/internal/_build/results?buildId=386178

Will not merge until 3.0.2.

[ryanbrandenburg]

Seems fine.

[dougbu]

Test official build corresponding to this PR is https://dev.azure.com/dnceng/internal/_build/results?buildId=418214&view=results. That was successful i.e. it didn't hit current issues with icon files nor previous issues with .msi and .cab files. Thanks @joeloff❕

[joeloff]

So we have to manually update the Arcade SDK version? I thought it flowed through the DARC process like other repos do

[dougbu]

I thought it flowed through the DARC process like other repos do

Kind-of and sort-of but not sufficiently. In particular, the Arcade SDK contains one or two versions files that are not updated automatically.