[Synapse] Make Synapse PowerShell consume track 2 Spark SDK #12202

wonner · 2020-06-18T05:41:25Z

Description

Make Synapse PowerShell consume track 2 Spark SDK.

Checklist

I have read the Submitting Changes section of CONTRIBUTING.md
The title of the PR is clear and informative
The appropriate ChangeLog.md file(s) has been updated:
- For any service, the ChangeLog.md file can be found at src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md
- A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header -- no new version header should be added
The PR does not introduce breaking changes
If applicable, the changes made in the PR have proper test coverage
For public API changes to cmdlets:
- a cmdlet design review was approved for the changes in this repository (Microsoft internal only)
- the markdown help files have been regenerated using the commands listed here

merge

adxsdkps · 2020-06-18T05:41:30Z

Can one of the admins verify this patch?

idear1203 · 2020-06-18T09:22:08Z

@VeryEarly @erich-wang We meet the following CI build error:

  Analyzer invoked with parameters: -p D:\a\1\s\artifacts/Debug -r D:\a\1\s\artifacts//StaticAnalysisResults -h -u -m
  No value provided for the --modules-to-analyze parameter. Filtering over all built modules.
  Analyzing module under D:\a\1\s\artifacts\Debug\Az.Accounts ...
  Analyzing module under D:\a\1\s\artifacts\Debug\Az.Storage ...
  Help record Set-AzDataLakeGen2ItemACLObject has no cmdlet.
  Analyzing module under D:\a\1\s\artifacts\Debug\Az.Synapse ...
  
  Unhandled Exception: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
  Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
  Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
  Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
     at Tools.Common.Loaders.CmdletLoader.GetModuleMetadata(String assemblyPath) in D:\a\1\s\tools\Tools.Common\Loaders\CmdletLoader.cs:line 209
     at Tools.Common.Loaders.CmdletLoader.GetModuleMetadata(String assemblyPath, List`1 commonOutputFolders) in D:\a\1\s\tools\Tools.Common\Loaders\CmdletLoader.cs:line 54
     at StaticAnalysis.HelpAnalyzer.HelpAnalyzer.AnalyzeMarkdownHelp(IEnumerable`1 scopes, String directory, ReportLogger`1 helpLogger, List`1 processedHelpFiles, String savedDirectory) in D:\a\1\s\tools\StaticAnalysis\HelpAnalyzer\HelpAnalyzer.cs:line 279
     at StaticAnalysis.HelpAnalyzer.HelpAnalyzer.Analyze(IEnumerable`1 scopes, IEnumerable`1 modulesToAnalyze) in D:\a\1\s\tools\StaticAnalysis\HelpAnalyzer\HelpAnalyzer.cs:line 111
     at StaticAnalysis.Program.Main(String[] args) in D:\a\1\s\tools\StaticAnalysis\Program.cs:line 139
D:\a\1\s\build.proj(208,5): error MSB3073: The command "dotnet D:\a\1\s\artifacts/StaticAnalysis/StaticAnalysis.Netcore.dll -p D:\a\1\s\artifacts/Debug -r D:\a\1\s\artifacts//StaticAnalysisResults -h -u -m " exited with code -532462766.
##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

We did some investigation locally. It seems that there are some issues when both Az.Synapse and Az.Storage exist under the artifacts\Debug folder.

If I remove the Az.Storage from artifacts\Debug folder, the command works fine:

PS> dotnet D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysis\StaticAnalysis.Netcore.dll -p D:\code\AzureSDK\azure-powershell\artifacts\Debug -r D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysisResults -h -u -m
Analyzer invoked with parameters: -p D:\code\AzureSDK\azure-powershell\artifacts\Debug -r D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysisResults -h -u -m
No value provided for the --modules-to-analyze parameter. Filtering over all built modules.
Analyzing module under D:\code\AzureSDK\azure-powershell\artifacts\Debug\Az.Accounts ...
Analyzing module under D:\code\AzureSDK\azure-powershell\artifacts\Debug\Az.Synapse ...
PS>

But the command will fail if Az.Storage exist.

PS> dotnet D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysis\StaticAnalysis.Netcore.dll -p D:\code\AzureSDK\azure-powershell\artifacts\Debug -r D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysisResults -h -u -m
Analyzer invoked with parameters: -p D:\code\AzureSDK\azure-powershell\artifacts\Debug -r D:\code\AzureSDK\azure-powershell\artifacts\StaticAnalysisResults -h -u -m
No value provided for the --modules-to-analyze parameter. Filtering over all built modules.
Analyzing module under D:\code\AzureSDK\azure-powershell\artifacts\Debug\Az.Accounts ...
Analyzing module under D:\code\AzureSDK\azure-powershell\artifacts\Debug\Az.Storage ...
Help record Set-AzDataLakeGen2ItemACLObject has no cmdlet.
Analyzing module under D:\code\AzureSDK\azure-powershell\artifacts\Debug\Az.Synapse ...

Unhandled Exception: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
Could not load file or assembly 'Azure.Core, Version=1.2.2.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)
   at StaticAnalysis.Program.Main(String[] args) in D:\code\AzureSDK\azure-powershell\tools\StaticAnalysis\Program.cs:line 149

Is it because the Synapse project references the Storage project? Could you please provide suggestions about how we should fix the issue? Thanks in advance.

erich-wang · 2020-06-22T08:49:02Z

src/Synapse/Synapse/Synapse.csproj

+	<Target Name="CopyFiles" AfterTargets="Build">
+		<Copy SourceFiles="@(PreLoadAssemblies)" DestinationFolder="$(TargetDir)\PreloadAssemblies" />
+		<Copy SourceFiles="@(NetCoreAssemblies)" DestinationFolder="$(TargetDir)\NetCoreAssemblies" />
+	</Target>
+


Please remove these lines

update

erich-wang · 2020-06-29T06:34:57Z

/azp run

azure-pipelines · 2020-06-29T06:35:15Z

Azure Pipelines successfully started running 3 pipeline(s).

erich-wang

It seems there's some conflict between this PR and latest master branch, please rebase this PR based on latest master.

erich-wang · 2020-06-29T07:01:45Z

src/Synapse/Synapse/Common/AzureSessionCredential.cs

+        private IAzureEnvironment EnsureSynapseOAuthAudienceSet(IAzureEnvironment environment)
+        {
+            if (environment != null)
+            {
+                if (!environment.IsPropertySet(SynapseOAuthEndpointResourceKey))
+                {
+                    environment.SetProperty(SynapseOAuthEndpointResourceKey, SynapseOAuthEndpointResourceValue);
+                }
+            }
+
+            return environment;
+        }


The environment related change should be in another repo https://github.com/Azure/azure-powershell-common, you may search related code for https://github.com/Azure/azure-powershell-common/blob/master/src/Authentication.Abstractions/AzureEnvironmentConstants.cs#L118..L121 as reference.
BTW, does synapse have same endpoint for different azure cloud environments? E.g. UsGov, GermanGov environments.

@erich-wang , we already added environment related config in azure-powershell-common: https://github.com/Azure/azure-powershell-common/blob/master/src/Authentication.Abstractions/AzureEnvironmentConstants.cs#L194-L195 . We added Synapse related endpoints as extended properties. But this comment makes sense to me. @wonner , could you please try removing this method and authenticate as below (line 27)?

IAccessToken accessToken1 = AzureSession.Instance.AuthenticationFactory.Authenticate( DefaultContext.Account, DefaultContext.Environment, DefaultContext.Tenant.Id, null, ShowDialog.Never, null, AzureEnvironment.ExtendedEndpoint.AzureSynapseAnalyticsEndpointResourceId);

@erich-wang for your 2nd question. Currently Synapse only have endpoints for Global cloud and China cloud. Other national clouds are not supported.

update

idear1203 · 2020-06-29T08:23:22Z

src/Synapse/Synapse/Common/AzureSessionCredential.cs

+        /// <summary>
+        /// Default resourceId for synapse OAuth tokens
+        /// </summary>
+        public const string SynapseOAuthEndpointResourceValue = "https://dev.azuresynapse.net";


Do we still need these two constants?

idear1203 · 2020-06-29T09:46:27Z

/azp run

azure-pipelines · 2020-06-29T09:46:44Z

Azure Pipelines successfully started running 3 pipeline(s).

erich-wang · 2020-06-30T01:38:37Z

/azp run

azure-pipelines · 2020-06-30T01:38:54Z

Azure Pipelines successfully started running 3 pipeline(s).

idear1203 · 2020-06-30T12:30:07Z

src/Synapse/Synapse/Synapse.csproj

@@ -12,7 +12,7 @@

  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Management.Synapse" Version="0.1.0-preview.2" />
-    <PackageReference Include="Microsoft.Azure.Synapse" Version="0.1.0-preview" />
+	<PackageReference Include="Azure.Analytics.Synapse.Spark" Version="1.0.0-preview.1" />


tab to spaces?

src/Synapse/Synapse/Az.Synapse.psd1

idear1203

LGTM. @erich-wang @VeryEarly could you please take a look at this PR again from your end?

The base branch was changed.

) * update NewAzureSynapseSparkPool * fix NewAzureSynapseSparkPool * fix NewAzureSynapseSparkPool for auto scale * fix New-AzSynapseSparkPool for auto scale * add New-AzSynapseSparkPool help doc * fix New-AzSynapseSparkPool help doc * update parameter set name * track2 v1.0 * track2 v2.0 * track2 v2.1 * Make Synapse PowerShell consume track 2 SDK * Make Synapse PowerShell consume track 2 SDK * add change log * fix proj * Make Synapse PowerShell consume track 2 SDK * fix AzureSessionCredential * update changelog * fix AzureSessionCredential * Add required assembly Azure.Analytics.Synapse.Spark.dll to psd1 * change tab to space Co-authored-by: Wan Yang <[email protected]> Co-authored-by: erich-wang <[email protected]>

Wan Yang and others added 16 commits June 10, 2020 10:41

update NewAzureSynapseSparkPool

2b5e92d

fix NewAzureSynapseSparkPool

4832c5f

fix NewAzureSynapseSparkPool for auto scale

0b3f6d6

fix New-AzSynapseSparkPool for auto scale

7443a56

add New-AzSynapseSparkPool help doc

3d7312d

fix New-AzSynapseSparkPool help doc

cfd848a

update parameter set name

fc580c9

track2 v1.0

6df7c14

track2 v2.0

50d9f0f

track2 v2.1

201c8e8

Make Synapse PowerShell consume track 2 SDK

050da0b

Merge pull request #1 from Azure/master

6d9a86e

merge

Merge branch 'master' of https://github.com/wonner/azure-powershell

f737ca1

Merge branch 'dev'

97013ee

Make Synapse PowerShell consume track 2 SDK

3a22af5

add change log

789e715

wonner requested a review from idear1203 June 18, 2020 05:41

fix proj

e676984

VeryEarly self-assigned this Jun 18, 2020

erich-wang self-assigned this Jun 18, 2020

azuresdkci added the needs-review label Jun 19, 2020

erich-wang reviewed Jun 22, 2020

View reviewed changes

Wan Yang and others added 4 commits June 24, 2020 13:38

Make Synapse PowerShell consume track 2 SDK

a91dbd0

Merge pull request #2 from Azure/master

f212bbb

update

Merge pull request #3 from Azure/master

26f3c2f

update

Merge branch 'master' of https://github.com/wonner/azure-powershell

7a2bf17

erich-wang suggested changes Jun 29, 2020

View reviewed changes

wonner and others added 4 commits June 29, 2020 16:10

Merge pull request #4 from Azure/master

5146c60

update

fix AzureSessionCredential

e673732

Merge branch 'master' of https://github.com/wonner/azure-powershell

b3dbb7a

update changelog

0b20221

idear1203 reviewed Jun 29, 2020

View reviewed changes

fix AzureSessionCredential

0f71eca

erich-wang changed the base branch from master to erich/core-static-analysis June 30, 2020 09:03

Add required assembly Azure.Analytics.Synapse.Spark.dll to psd1

1513079

idear1203 reviewed Jun 30, 2020

View reviewed changes

Wan Yang added 2 commits July 1, 2020 10:02

change tab to space

ee78736

Merge branch 'yw-dev' of https://github.com/wonner/azure-powershell

664c656

idear1203 previously approved these changes Jul 1, 2020

View reviewed changes

erich-wang changed the base branch from erich/core-static-analysis to master July 1, 2020 07:52

erich-wang approved these changes Jul 1, 2020

View reviewed changes

erich-wang merged commit 598cef5 into Azure:master Jul 1, 2020

[Synapse] Make Synapse PowerShell consume track 2 Spark SDK #12202

[Synapse] Make Synapse PowerShell consume track 2 Spark SDK #12202

Uh oh!

Conversation

wonner commented Jun 18, 2020

Description

Checklist

Uh oh!

adxsdkps commented Jun 18, 2020

Uh oh!

idear1203 commented Jun 18, 2020

Uh oh!

erich-wang Jun 22, 2020

Choose a reason for hiding this comment

Uh oh!

erich-wang commented Jun 29, 2020

Uh oh!

azure-pipelines bot commented Jun 29, 2020

Uh oh!

erich-wang left a comment

Choose a reason for hiding this comment

Uh oh!

erich-wang Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

idear1203 Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

idear1203 Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

idear1203 Jun 29, 2020

Choose a reason for hiding this comment

Uh oh!

idear1203 commented Jun 29, 2020

Uh oh!

azure-pipelines bot commented Jun 29, 2020

Uh oh!

erich-wang commented Jun 30, 2020

Uh oh!

azure-pipelines bot commented Jun 30, 2020

Uh oh!

idear1203 Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

idear1203 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!