[Storage] File cmdlet Track2, listblob track2

[blueww]

Description

Migrate file cmdlets to Track2 SDK, and list blob cmdlet to track2 SDK.
No cmdlet interface change, so no design doc needed

Add a static analysis breaking exception, since the "Get-AzStorageFileCopyState" cmdlet old return type "CopyState" and the new return type "PSCopyState" has exactly same child properties, so it should not be taken as breaking.

Checklist

• Check this box to confirm: I have read the Submitting Changes section of CONTRIBUTING.md and reviewed the following information:

• SHOULD select appropriate branch. Cmdlets from Autorest.PowerShell should go to generation branch.
• SHOULD make the title of PR clear and informative, and in the present imperative tense.
• SHOULD update ChangeLog.md file(s) appropriately
  • For any service, the ChangeLog.md file can be found at src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md
  • A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header in the past tense. Add changelog in description section if PR goes into generation branch.
  • Should not change ChangeLog.md if no new release is required, such as fixing test case only.
• SHOULD have approved design review for the changes in this repository (Microsoft internal only) with following situations
  • Create new module from scratch
  • Create new resource types which are not easy to conform to Azure PowerShell Design Guidelines
  • Create new resource type which name doesn't use module name as prefix
  • Have design question before implementation
• SHOULD regenerate markdown help files if there is cmdlet API change. Instruction
• SHOULD have proper test coverage for changes in pull request.
• SHOULD NOT introduce breaking changes in Az minor release except preview version.
• SHOULD NOT adjust version of module manually in pull request

[blueww]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[chandlerkent]

Hi @blueww, @wyunchi-ms, we recently had a production outage that I think I am able to track back to this change (in case you have access to it, we had opened Azure support case #2301120040007569).

I am just looking for confirmation for our customer-facing root cause analysis that this change would have caused the following behavior change:

If you load blobs with Get-AzStorageBlob (e.g. Get-AzStorageBlob -Container '$web' -Prefix 'assets/packages/js/' -Context $ctx) the ICloudBlob.Properties are no longer loaded by default until ICloudBlob.FetchAttributes() is invoked.

I confirmed that the behavior changed in version 4.10.0 of the Az.Storage powershell module through testing and I think this change had the unintended side effect that I described.

This caused us an issue particularly because it meant that scripts which set properties on Blobs would actually remove properties that were set by other scripts since none of the scripts were calling ICloudBlob.FetchAttributes(). The scenario is:

1. Script 1 sets $blob.ICloudBlob.Properties.ContentType and calls $blob.ICloudBlob.SetProperties() to persist
2. Script 2 sets $blob.ICloudBlob.Properties.CacheControl and calls $blob.ICloudBlob.SetProperties() to persist
3. Since script 2 did not call $blob.ICloudBlob.FetchAttributes() it removed ContentType which was set by script 1

Prior to these changes it was not required to call ICloudBlob.FetchAttributes().

[blueww]

@chandlerkent

Thanks for raising the issue!

On Az.Storage 5.0.0 (released on 2022-10-17), the Get-AzStorageBlob has the returned blob properties moved from ICloudBlob.Properties to BlobProperties (In latest Az.Storage module 5.3.0, for list multiple Blobs with Get-AzStorageBlob, you can also find the blob properties in ListBlobProperties, whose structure is a little different than BlobProperties.).

The change is caused to implement new storage features, we need to migrate Powershell cmdlets from v11 .net SDK to v12 .net SDK.

The "ICloudBlob.Properties.ContentType" matches to "ListBlobProperties.Properties.ContentType" (only for list blob) or "ListBlobProperties.Properties.ContentType".

The "ICloudBlob.Properties.CacheControl" matches to "ListBlobProperties.Properties.CacheControl" (only for list blob) or "ListBlobProperties.Properties.CacheControl".

If you still want to use properties in "ICloudBlob.Properties", please run "ICloudBlob.FetchAttributes()" before use it.

This is a breaking change, so we have followed the breaking change process:

the breaking change happens on a major release.
we have added warning message for the breaking change to the cmdlet in the before release, before the breaking change really happen. The warning message is added in #19340. So when you run Get-AzStorageBlob on a before release like Az.Storage 4.9.0, you will see warning like following :

WARNING: Upcoming breaking changes in the cmdlet 'Get-AzStorageBlob' :
The returned blob properties will be moved from ICloudBlob.Properties to BlobProperties in a future release.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

[chandlerkent]

@blueww thank you for the response!

Just FYI I actually think this broke in 4.10.0 based on my findings. Here is a screenshot showing the change:

The top-left is old AzureRM, top-right is 4.9.0, bottom-right is 4.10.0 where the behavior changed.

[blueww]

@dingmeng-xue

It looks Az.Storage 4.10.0 is released just 5 days before 5.0.0, and the breaking change PR is already merged in when it's released.
As I remember, there should not be a scheduled release on that time. Would you please clarify why we have Az.Storage 4.10.0 release?

@chandlerkent
Would you please apply the changes suggested in my before comments when use Az.Storage 4.10.0 or 5.0.0 or later?

[chandlerkent]

@blueww thank you for confirmation! We already applied the fix and can confirm it is working.

Since this caused our customers a production outage, we needed to provide a root cause, which you have helped us confirm.

Thanks again!