[Do Not Squash] merge latest main branch change to storage preview branch

.azure-pipelines/daily-build.yml

@@ -97,6 +97,8 @@ jobs:
         Release/**/Microsoft.Azure.PowerShell.Authentication.dll
         Release/**/Microsoft.Azure.PowerShell.Authentication.ResourceManager.dll
         Release/**/Microsoft.Azure.PowerShell.Authenticators.dll
+        Release/**/Microsoft.Azure.PowerShell.AuthenticationAssemblyLoadContext.dll
+        Release/**/Microsoft.Azure.Commands.*.AlcWrapper.dll
         !Release/**/Microsoft*Azure*PowerShell*Cmdlets*.Test.dll
       UseMinimatch: true
       signConfigType: 'inlineSignParams'

.azure-pipelines/util/sync-tools-folder-template.yml

@@ -13,6 +13,8 @@ steps:
       cp src/lib src/lib-tmp -r
 
       cp tools tools-tmp -r
+
+      cp build.proj build-tmp.proj
     pwsh: true
 - task: PowerShell@2
   displayName: Config git
@@ -25,6 +27,17 @@ steps:
 
       git checkout -b syncToolsFolder-${{ parameters.BranchName }} origin/${{ parameters.BranchName }}
     pwsh: true
+- task: PowerShell@2
+  displayName: Sync build.proj folder from main to ${{ parameters.BranchName }}
+  inputs:
+    targetType: inline
+    script: >-
+      rm build.proj
+
+      mv build-tmp.proj build.proj
+
+      git add build.proj
+    pwsh: true
 - task: PowerShell@2
   displayName: Sync .azure-pipelines folder from main to ${{ parameters.BranchName }}
   inputs:

.gitignore

@@ -235,3 +235,6 @@ launchSettings.json
 
 # Added due to scan
 src/DataFactory/DataFactoryV2.Test/SessionRecords/Microsoft.Azure.Commands.DataFactoryV2.Test.RunTests/TestRunV2.json
+
+# GitHub codespaces
+.venv

ChangeLog.md

@@ -1,3 +1,158 @@
+## 6.4.0 - September 2021
+#### Az.Accounts
+* Corrected the URLs to Azure Portal in the results of 'Get-AzEnvironment' and 'Get-AzContext'. [#15429]
+* Made infrastructural changes to support overriding default subscription via a '-SubscriptionId <String>' parameter.
+    - [Az.Aks](https://docs.microsoft.com/powershell/module/az.aks/get-azakscluster) is the first module that supports it.
+
+#### Az.Aks
+* Made '-Subscription <String>' available in all Aks cmdlets. You can manage Aks resources in other subscriptions without switching the context.
+
+#### Az.ApiManagement
+* Added new 'Sync-AzApiManagementKeyVaultSecret' cmdlet.
+* Added new 'New-AzApiManagementKeyVaultObject' cmdlet.
+* Added new optional [-useFromLocation] parameter to the 'Get-ApiManagementCache' 'New-ApiManagementCache''Update-ApiManagementCache' cmdlet.
+* Updated cmdlet **New-AzApiManagement** to manage ApiManagement service 
+    - Added support for the new 'Isolated' SKU
+    - Added support for managing Availability Zones using 'Zone' property
+    - Added support for Disabling Gateway in a Region using 'DisableGateway' property
+    - Added support for managing the minimum Api Version to allow for Control Plane using 'MinimalControlPlaneApiVersion' property.
+* Updated cmdlet **New-AzApiManagementRegion** to manage ApiManagement service     
+    - Added support for managing Availability Zones using 'Zone' property
+    - Added support for Disabling Gateway in a Region using 'DisableGateway' property
+* Updated cmdlet **Add-AzApiManagementRegion** to manage ApiManagement service     
+    - Added support for managing Availability Zones using 'Zone' property
+    - Added support for Disabling Gateway in a Region using 'DisableGateway' property
+* Updated cmdlet **Update-AzApiManagementRegion** to manage ApiManagement service     
+    - Added support for managing Availability Zones using 'Zone' property
+    - Added support for Disabling Gateway in a Region using 'DisableGateway' property
+* Updated cmdlet **New-AzApiManagementCustomHostnameConfiguration** to manage Custom Hostname Configuration
+    - Added support for specifying 'IdentityClientId' to provide Managed Identity User Assigned ClientId to use with KeyVault
+
+#### Az.Automation
+* Fixed bug: Closing in input file handle in Import-AzAutomationRunbook
+
+#### Az.Cdn
+* Fixed mandatory parameters issue in 'Get-AzCdnEndpointResourceUsage' cmdlet
+
+#### Az.Compute
+* Added new parameters '-LinuxConfigurationPatchMode', '-WindowsConfigurationPatchMode', and '-LinuxConfigurationProvisionVMAgent' to 'Set-AzVmssOSProfile'
+* Added new parameters '-SshKeyName' and '-GenerateSshKey' to 'New-AzVM' to create a VM with SSH
+* Fixed a bug in 'Add-AzVHD' on Linux that caused uploads to fail for certain destination URI
+* Added new cmdlets for Restore Points and Restore Point Collection:
+    - 'New-AzRestorePoint'
+    - 'New-AzRestorePointCollection'
+    - 'Get-AzRestorePoint'
+    - 'Get-AzRestorePointCollection'
+    - 'Update-AzRestorePointCollection'
+    - 'Remove-AzRestorePoint'
+    - 'Remove-AzRestorePointCollection'
+* Added new parameters '-EnableSpotRestore' and '-SpotRestoreTimeout' to 'New-AzVMSSConfig' to enable Spot Restore Policy 
+* Added new cmdlets: 'Update-AzCapacityReservationGroup' and 'Update-AzCapacityReservation'
+
+#### Az.CosmosDB
+* Fixed a bug where the restore of deleted database accounts fail.
+
+#### Az.DataFactory
+* Added a subnetId argument for the 'Set-AzDataFactoryV2IntegrationRuntime' cmdlet to support RBAC checking for VNet injection against the subnet resource ID instead of the VNet resource ID.
+* Added the 'Get-AzDataFactoryV2IntegrationRuntimeOutboundNetworkDependenciesEndpoint' cmdlet to provide a list of outbound network dependencies for SSIS integration runtime in Azure Data Factory that joins a virtual network.
+* Added PublicNetworkAccess to Data Factory.
+* Updated ADF .Net SDK version to 4.23.0
+
+#### Az.KeyVault
+* Supported adding EC keys in key vault [#15699]
+
+#### Az.Migrate
+* Supported duplicate disk UUID in source disk.
+* Supported subnets in same VNet for AVSet.
+* Supported runAsAccount fetching for multiple Vcenters in same site.
+
+#### Az.Network
+* Updated cmdlet to add 'Subnet' property for IP based load balancer backend address pool.
+    - 'New-AzLoadBalancerBackendAddressConfig'
+* Updated cmdlet to add 'TunnelInterface' property for backend pool related operations.
+    - 'New-AzLoadBalancerBackendAddressPool'
+    - 'Set-AzLoadBalancerBackendAddressPool'
+
+#### Az.RecoveryServices
+* Azure Site Recovery multi appliance support for VMware to Azure disaster recovery scenarios using RCM as the control plane.
+* Azure Backup fixed targetPhysicalPath issue with SQL CRR
+* Azure Backup fixed disable protection for SQL workload
+* Azure Backup resolved bug in setting CMK properties in latest release
+* Azure Backup removed special characters from register-azrecoveryservicesbackupcontainer command help text
+
+#### Az.Resources
+* Use JsonExtensions to serialize deserialize JSON objects to ensure the use of custom serialization settings [#15552]
+* Added support for 'Unsupported' and 'NoEffect' change types to deployment What-If cmdlets.
+
+#### Az.SecurityInsights
+* Updated to 'Get-AzSentinelIncident' parameters
+    - Added '-Filter' to support OData filter
+    - Added '-OrderBy' to support OData ordering
+    - Added '-Max' to support retrieving more than the default of 1000 incidents.
+
+#### Az.Sql
+* Changed the underlying implementation of 'Get-AzSqlDatabase' to support a paginated response from the server
+* Added 'ZoneRedundant' parameter to 'New-AzSqlInstance' and 'Set-AzSqlInstance' to enable the creation and the update of zone - redundant instances.
+* Added ZoneRedundant field to the model of the managed instance so that it displays information about zone - redundancy for instance that are returned by 'Get-AzSqlInstance'.
+* Extended AuditActionGroups enum in server & database audit. Added DBCC_GROUP, DATABASE_OWNERSHIP_CHANGE_GROUP and DATABASE_CHANGE_GROUP.
+* Added 'AsJob' flag to 'Remove-AzSqlInstance'
+* Added 'SubnetId' parameter to 'Set-AzSqlInstance' to support the cross-subnet update SLO
+* Upgraded to newest SDK version
+
+#### Az.Storage
+* Supported get/set blob tags on a specific blob
+    -  'Get-AzStorageBlobTag'
+    -  'Set-AzStorageBlobTag'
+* Supported create destination blob with specific blob tags while upload/copy Blob
+    -  'Set-AzStorageBlobContent'
+    -  'Start-AzStorageBlobCopy'
+* Supported list blobs across containers with a blob tag filter sql expression
+    -  'Get-AzStorageBlobByTag'
+* Supported list blobs inside a container and include Blob Tags
+    -  'Get-AzStorageBlob'
+* Supported run blob operation with blob tag condition, and fail the cmdlet when blob tag condition not match
+    -  'Get-AzStorageBlob'
+    -  'Get-AzStorageBlobContent'
+    -  'Get-AzStorageBlobTag'
+    -  'Remove-AzStorageBlob'
+    -  'Set-AzStorageBlobContent'
+    -  'Set-AzStorageBlobTag'
+    -  'Start-AzStorageBlobCopy'
+    -  'Stop-AzStorageBlobCopy'
+* Generate blob sas token with new API version
+    -  'New-AzStorageBlobSASToken' 
+    -  'New-AzStorageContainerSASToken' 
+    -  'New-AzStorageAccountSASToken'
+* Fixed blob copy failure with OAuth credential when client and server has time difference [#15644]
+    -  'Copy-AzStorageBlob' 
+* Fixed remove Data Lake Gen2 item fail with readonly SAS token
+    -  'Remove-AzDataLakeGen2Item' 
+* Revised destination existing check in move Data Lake Gen2 item
+    -  'Move-AzDataLakeGen2Item' 
+
+#### Az.StorageSync
+* Added parameter sets to 'Invoke-AzStorageSyncChangeDetection'
+    - Can call the cmdlet without -DirectoryPath and -Path parameters to trigger change detection on an entire file share
+* Added support for authoritative upload as part of New-AzStorageSyncServerEndpoint.
+* Added cloud change enumeration status information in Cloud Endpoint object.
+* Updated Server Endpoint object with various health properties
+* Added 'ServerName' property in Server Endpoint and Registered Server objects to support showing the current FQDN of a server.
+
+#### Az.Websites
+* Fixed 'Set-AzWebApp' to return a valid warning message when fails to add -Hostname #9316
+* Fixed 'Get-AzWebApp' to return CustomDomainVerificationId in the response. #9316
+
+### Thanks to our community contributors
+* Andrew Sears (@asears)
+  * Fix spelling of accountname (#15779)
+  * Fix Spelling, examples (#15780)
+* @cawrites, Update New-AzDataMigrationService.md (#15646)
+* @harpaul-gill, Adding support for pagination in Sql Get Databases (#15772)
+* @jeepingben, Create mutex names that are safe for Linux (fixes #15653) (#15666)
+* @LosManos, Docs: Parameter is ignored when listing secrets (#15788)
+* Mats Estensen (@matsest), docs: add examples for Update-AzSubscription (#15748)
+* Mauricio Arroyo (@mauricio-msft), Fix typo in cmdlet example (#15719)
+
 ## 6.3.0 - August 2021
 #### Az.Accounts
 * Disabled context auto saving when token cache persistence fails on Windows and macOS

build.proj

@@ -199,7 +199,7 @@
       <RuntimeDllsIncludeList>Microsoft.Powershell.*.dll,System*.dll,Microsoft.VisualBasic.dll,Microsoft.CSharp.dll,Microsoft.CodeAnalysis.dll,Microsoft.CodeAnalysis.CSharp.dll</RuntimeDllsIncludeList>
       <RuntimeDllsExcludeList>System.Security.Cryptography.ProtectedData.dll,System.Configuration.ConfigurationManager.dll,System.Runtime.CompilerServices.Unsafe.dll,System.IO.FileSystem.AccessControl.dll,System.Buffers.dll,System.Text.Encodings.Web.dll,System.CodeDom.dll,System.Management.dll,System.Text.Json.dll,System.Threading.Tasks.Extensions.dll</RuntimeDllsExcludeList>
     </PropertyGroup>
-    <Exec Command="$(PowerShellCoreCommandPrefix) &quot;Get-ChildItem -Path $(RepoArtifacts)/$(Configuration) -Recurse -Include $(RuntimeDllsIncludeList) -Exclude $(RuntimeDllsExcludeList) | Where-Object {$_.FullName -notlike '*PreloadAssemblies*' -and $_.FullName -notlike '*NetCoreAssemblies*'} | Remove-Item -Force&quot;"/>
+    <Exec Command="$(PowerShellCoreCommandPrefix) &quot;Get-ChildItem -Path $(RepoArtifacts)/$(Configuration) -Recurse -Include $(RuntimeDllsIncludeList) -Exclude $(RuntimeDllsExcludeList) | Where-Object {$_.FullName -notlike '*PreloadAssemblies*' -and $_.FullName -notlike '*NetCoreAssemblies*' -and $_.FullName -notlike '*AzSharedAlcAssemblies*' -and $_.FullName -notlike '*ModuleAlcAssemblies*'} | Remove-Item -Force&quot;"/>
     <Exec Command="$(PowerShellCoreCommandPrefix) &quot;Get-ChildItem -Path $(RepoArtifacts)/$(Configuration) -Recurse -Include 'runtimes' | Remove-Item -Recurse -Force&quot;" Condition="'$(CodeSign)' == 'true'" />
 
     <Exec Command="$(PowerShellCoreCommandPrefix) &quot;. $(RepoTools)/UpdateModules.ps1 -BuildConfig $(Configuration) -Scope $(Scope)&quot;"/>

documentation/development-docs/autogen-directives-for-azure-powershell.md

@@ -0,0 +1,8 @@
+# Autogen Directives for Azure Powershell
+## Directive Scenarios
+- [Resource Group Append](#Resource-Group-Append)
+### Resource Group Append
+To provide `ResourceGroupName` in returned object,  set `resourcegroup-append` as true in readme.md 
+```
+resourcegroup-append: true
+```

documentation/development-docs/enable-subscription-id-overriding.md

@@ -0,0 +1,102 @@
+# How to: Enable Overriding Subscription ID in Your Module
+
+- [Background](#background)
+- [Limitation](#limitation)
+- [Steps](#steps)
+  - [Add `SupportsSubscriptionId` Attribute](#add-supportssubscriptionid-attribute)
+  - [Regenerate Help Documents](#regenerate-help-documents)
+- [Notes / Troubleshooting](#notes--troubleshooting)
+  - [Your Cmdlet Implements `IDynamicParameters`](#your-cmdlet-implements-idynamicparameters)
+  - [Static Analysis Fails: Parameters are Removed](#static-analysis-fails-parameters-are-removed)
+
+## Background
+
+Working with **multiple Azure subscriptions** can be inconvenient in Azure PowerShell, because users has to keep switching the context, which is why we introduced a mechanism that can easily turn your module **multiple-subscription-friendly** in [Az v6.4.0](https://github.com/Azure/azure-powershell/blob/isra-fel-patch-1/ChangeLog.md#640---september-2021). For example
+
+```powershell
+# Legacy (switch subscription first)
+Select-AzSubscription -SubscriptionId "00000000-0000-0000-0000-000000000000"
+New-AzAksCluster ...
+
+# New (just -SubscriptionId)
+New-AzAksCluster ... -SubscriptionId "00000000-0000-0000-0000-000000000000"
+```
+
+The new design does not only simplify scripts, but also runs more efficiently, as it saves 1 cmdlet execution per subscription switching.
+
+## Limitation
+
+The feature was designed to balance between "supporting more login scenarios" and "being easy to use". Here are the limitations:
+
+- **One subscription, multiple accounts**: when you login Azure PowerShell with multiple user accounts, and there is one subscription owned by more than one of them, it is obvious that `-SubscriptionId` is not enough to tell which context you wish to use, but we do not want to introduce more parameters, so this is not supported.
+  - Work-around is to log in with only 1 user account.
+- **Management-plane only**: as subscription may not make as much sense in data-plane as in management-plane, this feature is suggested to be applied to management-plane cmdlets only.
+
+## Steps
+
+Here are the two simple steps to enable this for your module:
+
+### Add `SupportsSubscriptionId` Attribute
+
+You can either add it to the cmdlet base class, which applies to all the derived cmdlets; or to individual cmdlets.
+
+```csharp
+[SupportsSubscriptionId] // Adding to base class: all cmdlets that inherit `KubeCmdletBase` will benefit
+public abstract class KubeCmdletBase : AzureRMCmdlet { /* ... */ }
+
+[SupportsSubscriptionId] // Adding to cmdlet class: only this cmdlet will benefit
+public class NewAzureRmAks : CreateOrUpdateKubeBase { /* ... */ }
+```
+
+### Regenerate Help Documents
+
+By adding the attribute your cmdlet(s) get a `[-Subscription <String>]` parameter, so the help documents need to be regenerated.
+Please refer to [Azure PowerShell Help Generation](https://github.com/Azure/azure-powershell/blob/main/documentation/development-docs/help-generation.md) for more details.
+
+
+## Notes / Troubleshooting
+
+This design is compatible with most modules, however, there are some uncommon case you should be careful.
+
+### Your Cmdlet Implements `IDynamicParameters`
+
+The `-SubscriptionId` parameter is added via `IDynamicParameters` interface, so if your cmdlet has already implemented it, make sure:
+
+1. `GetDynamicParameters()` is decorated with [`new` modifier](https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/new-modifier).
+1. Call `base.GetDynamicParameters()` and combine the results in your `GetDynamicParameters()`.
+
+A sample implementation:
+
+```csharp
+public new object GetDynamicParameters()
+{
+    var parameters = base.GetDynamicParameters() as RuntimeDefinedParameterDictionary;
+    // here should be customized logic to construct `RuntimeDefinedParameter` objects,
+    // and call `parameters.Add()`
+    return parameters;
+}
+```
+
+### Static Analysis Fails: Parameters are Removed
+
+This happens when the constructor of your cmdlet throws an exception -- normally constructors are not called during static analysis, but things are different if this feature is enabled.
+Make sure you use `BeginProcessing()` for heavy work.
+
+```csharp
+// DO NOT
+public StorageSyncClientCmdletBase()
+{
+    InitializeComponent(); // do stuff that throws exception in test environment, like an API call
+}
+
+// DO
+public StorageSyncClientCmdletBase()
+{
+}
+
+protected override void BeginProcessing()
+{
+    base.BeginProcessing();
+    InitializeComponent();
+}
+```

documentation/internal/authinrdfe.md

@@ -0,0 +1,32 @@
+# Authentication and Authorization in RDFE (Azure) and ARM (Az/AzureRM)
+## RDFE Authorization
+In RDFE, users are authorized on a per-subscription basis. Users who are authorized to access a subscription may perform any action within that subscription - on the subscription itself, or on any resources in the subscription. There is no mechanism for limiting the access of an authorized user within a subscription.
+## RDFE Authentication Mechanisms
+- Management Certificate Authentication
+- User Authentication
+### RDFE Management Certificate Authentication
+Management certificate authentication is the most popular mechanism for authenticating RDFE calls for automation.  In this authentication mechanism, the public key of a management certificate is associated with one or more subscriptions.  Users in possession of the certificate private key use standard Http certificate authentication to negotiate an SSL session with the RDFE endpoint, and all subsequent calls in that session have access to any subscriptions associated with the certificate.
+
+To acquire management certificate credentials, you must download  a PublishingProfile from the portal (using Get-AzurePublishSettingsFile, or direct download from the portal), and import it using Import-AzurePublishSettingsFile.  This downloads a file containing management certificates for selected subscriptions. Importing will automatically add each certificate as an account.
+
+### RDFE User Authentication
+The Add-AzureAccount command can be used to acquire a token based on user credentials, and if the associated user is authorized for RDFE access to a subscription (they must be a classic administrator or co-admin of the subscription), they will have access to those subscriptions authorized to their account for classic administrator access
+
+## General Notes
+- Managemnt certificate authentication lasts for an entire TCP session.  User authentication is self-renewing, just as it is in ARM.
+
+## Programmatic Authentication for PowerShell clients
+- Clients are still authenticated using an IAuthenticationFactory, and the following overload:
+
+```c#
+SubscriptionCloudCredentials GetSubscriptionCloudCredentials(IAzureContext context)
+```
+Note that, the ARM token audience is different than the token audience used for RDFE, although the RDFE tolken audience is accepted by both endpoints.
+
+Similarly, RDFE clients can be created using the IClientFactory interface: 
+
+```c#
+TClient CreateClient<TClient>(IAzureContext context, string endpoint) where TClient : ServiceClient<TClient>;
+```
+
+Also note that, for management certificate authentication, authentication is performed as part of creating the http client - the ```ServiceClientCredentials``` returned above automatically apply the certificate to the http connection, which automatically performs HTTP certificate authentication using the certificate private key.  You can see this implementation here: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/mgmtcommon/ClientRuntime/ClientRuntime/CertificateCredentials.cs