ADL Updates to latest SDK in preparation for GA #3152

begoldsm · 2016-10-27T22:57:15Z

Description

This brings ADL powershell up to the latest SDK version in preparation for GA. The latest SDK is still preview, however it is our first GA candidate SDK and needs to be included in powershell to bring the following features:

Drastically improved download performance for distributed file and folder downloads
Better error handling and messaging for uploads and downloads
Full credential CRUD management, which is replacing Secret CRUD (secret CRUD is deprecated this release)
Full Firewall Rule CRUD
Full TrustedIdProvider CRUD
ADLS Account encryption support
Help improvements to better explain the difference between GET and LIST jobs
Get-AdlCatalogItem fixes for bad input.

This checklist is used to make sure that common guidelines for a pull request are followed. You can find a more complete discussion of PowerShell cmdlet best practices here.

I have read the contribution guidelines.
If changes were made to any cmdlet, the XML help was regenerated using the platyPSHelp module.
If any large changes are made to a service, they are reflected in the respective change log.

General Guidelines

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.
The pull request does not introduce breaking changes (unless a major version change occurs in the assembly and module).

Testing Guidelines

Pull request includes test coverage for the included changes.
PowerShell scripts used in tests should do any necessary setup as part of the test or suite setup, and should not use hard-coded values for locations or existing resources.

Cmdlet Signature Guidelines

New cmdlets that make changes or have side effects should implement ShouldProcess and have SupportShouldProcess=true specified in the cmdlet attribute. You can find more information on ShouldProcess here.
Cmdlet specifies OutputType attribute if any output is produced - if the cmdlet produces no output, it should implement a PassThrough parameter.

Cmdlet Parameter Guidelines

Parameter types should not expose types from the management library - complex parameter types should be defined in the module.
Complex parameter types are discouraged - a parameter type should be simple types as often as possible. If complex types are used, they should be shallow and easily creatable from a constructor or another cmdlet.
Cmdlet parameter sets should be mutually exclusive - each parameter set must have at least one mandatory parameter not in other parameter sets.

This enables us to test out a fix for download throttling scenarios. update the package to have more robust logic This gives more retries and better error messages.

Encryption Trusted ID providers Credential management Firewall rule management. Bug fixes for documentation. Updates for upload and download performance. Credential updates for internal testing. missed one file. add should process support for static analysis This makes static analysis happy and, for these cases, should be fine to include. update ga candidate private package and enable list credentials.

All packages are published now and should be pulled from nuget.

This will give users a heads up that all of this is changing/going away.

begoldsm · 2016-10-28T17:53:31Z

@markcowl is there anything else needed here to make sure this gets merged into the release?

markcowl · 2016-10-28T17:34:40Z

...nalytics/Commands.DataLakeAnalytics/Commands/NewAzureRmDataLakeAnalyticsCatalogCredential.cs

+
+namespace Microsoft.Azure.Commands.DataLakeAnalytics
+{
+    [Cmdlet(VerbsCommon.New, "AzureRmDataLakeAnalyticsCatalogCredential", SupportsShouldProcess = true), OutputType(typeof(USqlCredential))]


Since there are multiple parameter sets, need a default parameter set name

markcowl · 2016-10-28T17:35:42Z

...akeAnalytics/Commands.DataLakeAnalytics/Commands/NewAzureRmDataLakeAnalyticsCatalogSecret.cs

@@ -22,6 +22,7 @@ namespace Microsoft.Azure.Commands.DataLakeAnalytics
 {
    [Cmdlet(VerbsCommon.New, "AzureRmDataLakeAnalyticsCatalogSecret"), OutputType(typeof(USqlSecret))]
    [Alias("New-AdlCatalogSecret")]
+    [Obsolete("Catalog secrets are being deprecated in the next release. Please use New-AzureRmDataLakeAnalyticsCatalogCredential directly instead.")]


...in a future release...

markcowl · 2016-10-28T17:38:02Z

...ytics/Commands.DataLakeAnalytics/Commands/RemoveAzureRmDataLakeAnalyticsCatalogCredential.cs

+        [ValidateNotNull]
+        public PSCredential Password { get; set; }
+
+        [Parameter(Position = 4, Mandatory = false, HelpMessage = "Do not ask for confirmation.")]


Switch parameters should not be positional.

Also, do you need a Force parameter? If there aren't special conditions you want to prompt the user about, consider not having a Force parameter.

I maintain the force parameter to keep consistent with how the secret functionality prompts, for now, since it is the replacement. There shouldn't be an issue with removing it though, since credential removal doesn't have cascading ramifications. Thoughts there?

I just chose to remove it, no force required for remove credential.

markcowl · 2016-10-28T17:38:21Z

...Analytics/Commands.DataLakeAnalytics/Commands/RemoveAzureRmDataLakeAnalyticsCatalogSecret.cs

 using System.Management.Automation;

 namespace Microsoft.Azure.Commands.DataLakeAnalytics
 {
    [Cmdlet(VerbsCommon.Remove, "AzureRmDataLakeAnalyticsCatalogSecret", SupportsShouldProcess = true), OutputType(typeof(bool))]
    [Alias("Remove-AdlCatalogSecret")]
+    [Obsolete("Catalog secrets are being deprecated in the next release. Please use Remove-AzureRmDataLakeAnalyticsCatalogCredential directly instead.")]


... in a future release

markcowl · 2016-10-28T17:38:59Z

...nalytics/Commands.DataLakeAnalytics/Commands/SetAzureRmDataLakeAnalyticsCatalogCredential.cs

+
+namespace Microsoft.Azure.Commands.DataLakeAnalytics
+{ 
+    [Cmdlet(VerbsCommon.Set, "AzureRmDataLakeAnalyticsCatalogCredential", SupportsShouldProcess = true), OutputType(typeof(USqlCredential))]


Since there are multiple parameter sets, you should specify a default parameter set name

markcowl · 2016-10-28T17:49:36Z

...er/DataLakeStore/Commands.DataLakeStore/Commands/GetAzureRmDataLakeStoreTrustedIdProvider.cs

+
+namespace Microsoft.Azure.Commands.DataLakeStore
+{
+    [Cmdlet(VerbsCommon.Get, "AzureRmDataLakeStoreTrustedIdProvider"), OutputType(typeof(TrustedIdProvider), typeof(IList<TrustedIdProvider>))]


same comment on output types

markcowl · 2016-10-28T17:51:00Z

...ager/DataLakeStore/Commands.DataLakeStore/Commands/RemoveAzureRmDataLakeStoreFirewallRule.cs

+            HelpMessage =
+                "Indicates the delete should be immediately performed with no confirmation or prompting. Use carefully."
+            )]
+        public SwitchParameter Force { get; set; }


(1) switch parameters should not be positional

(2) You don't need a Force in this case - there are no special conditions you are prompting for

markcowl · 2016-10-28T17:51:46Z

...DataLakeStore/Commands.DataLakeStore/Commands/RemoveAzureRmDataLakeStoreTrustedIdProvider.cs

+            HelpMessage =
+                "Indicates the delete should be immediately performed with no confirmation or prompting. Use carefully."
+            )]
+        public SwitchParameter Force { get; set; }


switch parameters should not be positional

You don't need the Force parameter

markcowl · 2016-10-28T17:52:14Z

...Manager/DataLakeStore/Commands.DataLakeStore/Commands/SetAzureRmDataLakeStoreFirewallRule.cs

+
+namespace Microsoft.Azure.Commands.DataLakeStore
+{
+    [Cmdlet(VerbsCommon.Set, "AzureRmDataLakeStoreFirewallRule", SupportsShouldProcess = true), OutputType(typeof(FirewallRule))]


same comment on output type

markcowl · 2016-10-28T17:52:54Z

...er/DataLakeStore/Commands.DataLakeStore/Commands/SetAzureRmDataLakeStoreTrustedIdProvider.cs

+
+namespace Microsoft.Azure.Commands.DataLakeStore
+{
+    [Cmdlet(VerbsCommon.Set, "AzureRmDataLakeStoreTrustedIdProvider", SupportsShouldProcess = true), OutputType(typeof(TrustedIdProvider))]


same comment on output types

cormacpayne · 2016-10-28T21:52:55Z

On-demand: http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1279/

begoldsm added 9 commits October 26, 2016 21:24

Update to latest private ADLS package

4cfb92b

This enables us to test out a fix for download throttling scenarios. update the package to have more robust logic This gives more retries and better error messages.

Add deprecation warnings for secret management.

2f778bb

Add tests and re-run to ensure all are passing.

b121cb9

Add changes to the change log.

55a7edb

Support Should Process for new cmdlets.

e3ba6db

Rerun filesystem tests and include new ADL tests

2b284f1

Remove Local feed for ADL

b6b6dcc

All packages are published now and should be pulled from nuget.

Add obsolete warning for the old credential object

282b3a6

This will give users a heads up that all of this is changing/going away.

azurecla added the cla-required label Oct 27, 2016

markcowl self-assigned this Oct 27, 2016

markcowl added cla-not-required and removed cla-required labels Oct 27, 2016

markcowl requested changes Oct 28, 2016

View reviewed changes

begoldsm added 5 commits October 28, 2016 12:09

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

a443f27

Address PR feedback.

d643911

Double check help, missed a force remove

3af7b46

Properly update types for GET

2532e3e

Accidentally unchecked in last commit for types

9831043

markcowl approved these changes Oct 28, 2016

View reviewed changes

markcowl merged commit 8d62464 into Azure:release-3.1.0 Oct 28, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ADL Updates to latest SDK in preparation for GA #3152

ADL Updates to latest SDK in preparation for GA #3152

Uh oh!

begoldsm commented Oct 27, 2016 •

edited

Loading

Uh oh!

begoldsm commented Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

begoldsm Oct 28, 2016

Uh oh!

begoldsm Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

markcowl Oct 28, 2016

Uh oh!

cormacpayne commented Oct 28, 2016

Uh oh!

Uh oh!

ADL Updates to latest SDK in preparation for GA #3152

ADL Updates to latest SDK in preparation for GA #3152

Uh oh!

Conversation

begoldsm commented Oct 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

begoldsm commented Oct 28, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cormacpayne commented Oct 28, 2016

Uh oh!

Uh oh!

begoldsm commented Oct 27, 2016 •

edited

Loading