[Synapse] Support for setting spark pool configuration by artifact

[yanjungao718]

Updated New-AzSynapseSparkPool and Update-AzSynapseSparkPool to support for setting spark pool configuration artifact by -SparkCongifuration, parameter -SparkConfigFilePath is deprecated now.

Checklist

• Check this box to confirm: I have read the Submitting Changes section of CONTRIBUTING.md and reviewed the following information:

• SHOULD select appropriate branch. Cmdlets from Autorest.PowerShell should go to generation branch.
• SHOULD make the title of PR clear and informative, and in the present imperative tense.
• SHOULD update ChangeLog.md file(s) appropriately
  • For any service, the ChangeLog.md file can be found at src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md
  • A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header in the past tense. Add changelog in description section if PR goes into generation branch.
  • Should not change ChangeLog.md if no new release is required, such as fixing test case only.
• SHOULD have approved design review for the changes in this repository (Microsoft internal only) with following situations
  • Create new module from scratch
  • Create new resource types which are not easy to conform to Azure PowerShell Design Guidelines
  • Create new resource type which name doesn't use module name as prefix
  • Have design question before implementation
• SHOULD regenerate markdown help files if there is cmdlet API change. Instruction
• SHOULD have proper test coverage for changes in pull request.
• SHOULD NOT introduce breaking changes in Az minor release except preview version.
• SHOULD NOT adjust version of module manually in pull request

[wyunchi-ms]

/azp run azure-powershell - security-tools

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Nickcandy]

It looks good to me. Please help me double check @BethanyZhou

[Nickcandy]

/azp run azure-powershell - security-tools

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[BethanyZhou]

What do you mean parameter -SparkConfigFilePath is deprecated now? Does this parameter not work anymore? Will current change will break previous user experience?

[yanjungao718]

Specifying spark pool configuration using file path in Synapse Studio and Azure Portal is deprecated, now we can only make it through spark configuration artifact, so add a new parameter to support it.
The old parameter -SparkConfigFilePath is useless now, it won't take any effect for updating spark pool.

[BethanyZhou]

I guess in our next major release, SparkConfigFilePath should be removed from these cmdlets, am I correct? If so, please add upcoming breaking change message for this parameter

What happens if user uses SparkConfigFilePath in their current scripts? Will these scripts break now? If so, seems like it's a breaking change from service team.

[yanjungao718]

I guess in our next major release, SparkConfigFilePath should be removed from these cmdlets, am I correct? If so, please add upcoming breaking change message for this parameter
yes, SparkConfigFilePath should be removed in future.
What happens if user uses SparkConfigFilePath in their current scripts? Will these scripts break now? If so, seems like it's a breaking change from service team.
if user specify -SparkConfigFilePath parameter in their current scripts, they will not break.

[BethanyZhou]

• Let's follow https://eng.ms/docs/cloud-ai-platform/azure-core/azure-management-and-platforms/control-plane-bburns/azure-cli-tools-azure-cli-powershell-and-terraform/azure-cli-tools/teams_docs/azps_docs/breakingchange_release_process#how-to-releae-a-breaking-change to pre-announce the deprecation of SparkConfigFilePath
• Need to confirm when SparkConfigFilePath will not work for customer, it may still be regarded as a breaking change.

[yanjungao718]

• Let's follow https://eng.ms/docs/cloud-ai-platform/azure-core/azure-management-and-platforms/control-plane-bburns/azure-cli-tools-azure-cli-powershell-and-terraform/azure-cli-tools/teams_docs/azps_docs/breakingchange_release_process#how-to-releae-a-breaking-change to pre-announce the deprecation of SparkConfigFilePath
  Added the breaking change notifications for existing cmdlets parameter.

• Need to confirm when SparkConfigFilePath will not work for customer, it may still be regarded as a breaking change.
  Confirmed, the SparkConfigFilePath may stop working until middle of this December.

[BethanyZhou]

Looks good to me with minor fix required

[BethanyZhou]

Looks good to me. My last question is is possible to transfer SparkConfigFilePath to SparkConfiguration? If so, maybe we can hide the difference between them.

[BethanyZhou]

/azp run azure-powershell - security-tools

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).