[Data Factory] Add data flow and debug session cmdlets #10177

jackmagic313 · 2019-10-03T01:38:54Z

Description

Design review tickets:
https://github.com/Azure/azure-powershell-cmdlet-review-pr/issues/388
https://github.com/Azure/azure-powershell-cmdlet-review-pr/issues/390

Changes:

Adding CRUD commands for ADF V2 data flow: Set-AzDataFactoryV2DataFlow, Remove-AzDataFactoryV2DataFlow, and Get-AzDataFactoryV2DataFlow.
Adding action commands for ADF V2 data flow debug Session: Start-AzDataFactoryV2DataFlowDebugSession, Get-AzDataFactoryV2DataFlowDebugSession, Add-AzDataFactoryV2DataFlowDebugSessionPackage, Invoke-AzDataFactoryV2DataFlowDebugSessionCommand and Stop-AzDataFactoryV2DataFlowDebugSession.
Update ADF .Net SDK version to 4.2.0

Checklist

I have read the Submitting Changes section of CONTRIBUTING.md
The title of the PR is clear and informative
The appropriate ChangeLog.md file(s) has been updated:
- For any service, the ChangeLog.md file can be found at src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md
- A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header -- no new version header should be added
The PR does not introduce breaking changes
If applicable, the changes made in the PR have proper test coverage
For public API changes to cmdlets:
- a cmdlet design review was approved for the changes in this repository (Microsoft internal only)
- the markdown help files have been regenerated using the commands listed here

…to jikma/adddataflow

markcowl · 2019-10-03T01:58:14Z

...DataFactoryV2/DataFlowDebugSessions/AddAzureDataFactoryDataFlowDebugSessionPackageCommand.cs

+                package.SessionId = SessionId;
+            }
+
+            DataFactoryClient.AddDataFlowToDebugSession(ResourceGroupName, DataFactoryName, package);


Need to actually implement ShouldProcess, see: https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/design-guidelines/should-process-confirm-impact.md

markcowl · 2019-10-03T01:59:32Z

...aFactoryV2/DataFlowDebugSessions/InvokeAzureDataFactoryDataFlowDebugSessionCommandCommand.cs

+            HelpMessage = Constants.HelpDataFlowDebugRowLimits)]
+        [Parameter(ParameterSetName = ParameterSetNames.ByResourceId, Position = 4, Mandatory = false,
+            HelpMessage = Constants.HelpDataFlowDebugRowLimits)]
+        public int? RowLimits { get; set; }


markcowl · 2019-10-03T01:59:42Z

...aFactoryV2/DataFlowDebugSessions/InvokeAzureDataFactoryDataFlowDebugSessionCommandCommand.cs

+            HelpMessage = Constants.HelpDataFlowDebugColumns)]
+        [Parameter(ParameterSetName = ParameterSetNames.ByResourceId, Position = 6, Mandatory = false,
+            HelpMessage = Constants.HelpDataFlowDebugColumns)]
+        public List<string> Columns { get; set; }


markcowl · 2019-10-03T02:00:18Z

...aFactoryV2/DataFlowDebugSessions/InvokeAzureDataFactoryDataFlowDebugSessionCommandCommand.cs

+                }
+
+            };
+            WriteObject(DataFactoryClient.InvokeDataFlowDebugSessionCommand(ResourceGroupName, DataFactoryName, request));


Needs to actually implement ShouldProcess

markcowl · 2019-10-03T02:01:50Z

...tory/DataFactoryV2/DataFlowDebugSessions/StartAzureDataFactoryDataFlowDebugSessionCommand.cs

+                request.IntegrationRuntime = integrationRuntimeResource;
+            }
+
+            WriteObject(DataFactoryClient.StartDebugSession(ResourceGroupName, DataFactoryName, request));


ShouldProcess implementation required

markcowl · 2019-10-03T02:08:39Z

src/DataFactory/DataFactoryV2/help/Remove-AzDataFactoryV2DataFlow.md

+---
+external help file: Microsoft.Azure.PowerShell.Cmdlets.DataFactoryV2.dll-Help.xml
+Module Name: Az.DataFactory
+online version:


same comment

markcowl · 2019-10-03T02:08:52Z

src/DataFactory/DataFactoryV2/help/Set-AzDataFactoryV2DataFlow.md

+---
+external help file: Microsoft.Azure.PowerShell.Cmdlets.DataFactoryV2.dll-Help.xml
+Module Name: Az.DataFactory
+online version:


same comment

markcowl · 2019-10-03T02:09:04Z

src/DataFactory/DataFactoryV2/help/Set-AzDataFactoryV2DataFlow.md

+```powershell
+PS C:\> Set-AzDataFactoryV2DataFlow -ResourceGroupName "ADF" -DataFactoryName "UncycloADF" -Name "TaxiDemo1" -DefinitionFile "C:\\samples\\UncycloSample\\TaxiDemo1.json"
+
+DataFlowName      : TaxiDemo1


table format?

this is list format

markcowl · 2019-10-03T02:09:17Z

src/DataFactory/DataFactoryV2/help/Start-AzDataFactoryV2DataFlowDebugSession.md

+---
+external help file: Microsoft.Azure.PowerShell.Cmdlets.DataFactoryV2.dll-Help.xml
+Module Name: Az.DataFactory
+online version:


need online version

markcowl · 2019-10-03T02:09:34Z

src/DataFactory/DataFactoryV2/help/Stop-AzDataFactoryV2DataFlowDebugSession.md

+---
+external help file: Microsoft.Azure.PowerShell.Cmdlets.DataFactoryV2.dll-Help.xml
+Module Name: Az.DataFactory
+online version:


online version

jackmagic313 · 2019-10-04T00:33:25Z

Thanks @markcowl for the quick review! I just addressed the feedbacks.

markcowl · 2019-10-04T12:36:19Z

...ctory/DataFactoryV2/DataFlowDebugSessions/StopAzureDataFactoryDataFlowDebugSessionCommand.cs

+{
+    [Cmdlet("Stop", ResourceManager.Common.AzureRMConstants.AzureRMPrefix + "DataFactoryV2DataFlowDebugSession", DefaultParameterSetName = ParameterSetNames.ByFactoryName, SupportsShouldProcess = true)]
+    [OutputType(typeof(void))]
+    [OutputType(typeof(string))]


It outputs bool

not sure what I did :P it's fixed.

markcowl · 2019-10-04T12:36:54Z

src/DataFactory/DataFactoryV2/DataFlows/RemoveAzureDataFactoryDataFlowCommand.cs

+namespace Microsoft.Azure.Commands.DataFactoryV2
+{
+    [Cmdlet("Remove", ResourceManager.Common.AzureRMConstants.AzureRMPrefix + "DataFactoryV2DataFlow", DefaultParameterSetName = ParameterSetNames.ByFactoryName,SupportsShouldProcess = true), OutputType(typeof(void))]
+    public class RemoveAzureDataFactoryDataFlowCommand : DataFactoryContextActionBaseCmdlet


Why is this returning string?

markcowl · 2019-10-04T12:38:17Z

src/DataFactory/DataFactoryV2/help/Get-AzDataFactoryV2DataFlow.md

+```powershell
+PS C:\> Get-AzDataFactoryV2DataFlow -ResourceGroupName "ADF" -DataFactoryName "UncycloADF"
+
+DataFlowName      : TaxiDemo1


This is list format

markcowl · 2019-10-04T12:39:22Z

src/DataFactory/DataFactoryV2/help/Get-AzDataFactoryV2DataFlow.md

+```powershell
+PS C:\> Get-AzDataFactoryV2DataFlow -ResourceGroupName "ADF" -DataFactoryName "UncycloADF" -Name "dataflow1"
+
+DataFlowName      : dataflow1


list format. You will need to have a format entry for this in order to have a table format, see: https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/generate-format.ps1xml-file.md

Oh I see. I thought I need to include a example to show how to convert to a table format result 😅, will update my pr soon

The generation tool doesn't work with "Could not load file or assembly 'System.Runtime.Loader, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified." error (I tried to change the version to 4.0.0 in csproj and I got a different error with something about assembly context). For now I just added manually into format file instead and it should still work if someone uses the script to generate the file in the future.

markcowl · 2019-10-04T12:39:39Z

src/DataFactory/DataFactoryV2/help/Get-AzDataFactoryV2DataFlowDebugSession.md

+```powershell
+PS C:\WINDOWS\system32> Get-AzDataFactoryV2DataFlowDebugSession -ResourceGroupName adf -DataFactoryName UncycloADF
+
+DataFlowName           : DebugSession-0a7e0d6e-f2b7-48cc-8cd8-618326f5662f


still lidt format

markcowl · 2019-10-04T12:40:11Z

src/DataFactory/DataFactoryV2/help/Invoke-AzDataFactoryV2DataFlowDebugSessionCommand.md

+
+Id     Name            PSJobTypeName   State         HasMoreData     Location             Command
+--     ----            -------------   -----         -----------     --------             -------
+3      Long Running... AzureLongRun... Running       True            localhost            Invoke-AzDataFactoryV2...


(this is table format)

markcowl · 2019-10-04T12:40:53Z

src/DataFactory/DataFactoryV2/help/Set-AzDataFactoryV2DataFlow.md

+```powershell
+PS C:\> Set-AzDataFactoryV2DataFlow -ResourceGroupName "ADF" -DataFactoryName "UncycloADF" -Name "TaxiDemo1" -DefinitionFile "C:\\samples\\UncycloSample\\TaxiDemo1.json"
+
+DataFlowName      : TaxiDemo1


this is list format

Add data flow cmdlets

92c62dc

azuresdkci added the needs-review label Oct 3, 2019

Merge branch 'master' of https://github.com/Azure/azure-powershell in…

85895e5

…to jikma/adddataflow

markcowl requested changes Oct 3, 2019

View reviewed changes

markcowl added needs-revision and removed needs-review labels Oct 3, 2019

jackmagic313 added 2 commits October 3, 2019 15:02

address feedback

5484d9e

Remove code indicator since security check doesn't like it

621bea1

jackmagic313 requested a review from markcowl October 4, 2019 00:33

markcowl requested changes Oct 4, 2019

View reviewed changes

markcowl assigned jackmagic313 Oct 4, 2019

address feedback

cdbd88d

jackmagic313 requested a review from markcowl October 4, 2019 20:25

markcowl previously approved these changes Oct 7, 2019

View reviewed changes

markcowl added Waiting for CI and removed needs-revision labels Oct 7, 2019

use to trigger the build again

efa9ce9

jackmagic313 dismissed markcowl’s stale review via efa9ce9 October 7, 2019 20:27

Merge branch 'master' into jikma/adddataflow

bf48ca8

erich-wang added the Sprint Candidate label Oct 8, 2019

markcowl approved these changes Oct 8, 2019

View reviewed changes

markcowl merged commit 3b131a3 into Azure:master Oct 8, 2019

msJinLei mentioned this pull request Oct 9, 2019

Static analysis in CI fails to identify the positioning parameter count exceeding issue #10221

Closed

[Data Factory] Add data flow and debug session cmdlets #10177

[Data Factory] Add data flow and debug session cmdlets #10177

Uh oh!

Conversation

jackmagic313 commented Oct 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackmagic313 commented Oct 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jackmagic313 commented Oct 3, 2019 •

edited

Loading