ADL Updates to latest Package #2892

begoldsm · 2016-09-03T00:49:30Z

Comments

Fix folder upload/download bug that would result in a hang
Fix crashes in progress tracking for all uploads/downloads
Fix crash in retrieving file information with malformed dates
Add Support for ACL and Default ACL removal
Add initial framework support for credential management
Enable friendly ACL display
Return more useful information with file properties (expiration time) and job (job statistics improvements for job finalization).
Add support for Getting and Setting unnamed permissions on files and folders.

9. Add support for getting information about account encryption states (the ability to encrypt will come in a future release).

This checklist is used to make sure that common issues in a pull request are covered by the creator. You can find a more complete discussion of PowerShell cmdlet best practices here.

Below in Overall Changes, check off the boxes that apply to your PR. For the categories that you did not check off, you can remove them from this body. Within each of the categories that you did select, make sure that you can check off all of the boxes.

For information on cleaning up the commits in your pull request, click here.

General

Title of the PR is clear and informative
There are a small number of commits that each have an informative message
If it applies, references the bug/issue that the PR fixes
All files have the Microsoft copyright header
Cmdlets refer to management libraries through nuget references - no dlls are checked in
The PR does not introduce breaking changes (unless a major version change occurs in the assembly and module)

Tests

PR includes test coverage for the included changes
Tests must use xunit, and should either use Moq to mock management client calls, or use the scenario test framework
PowerShell scripts used in tests must not use hard-coded values for location
PowerShell scripts used in tests should do any necessary setup as part of the test or suite setup, and should not use hard-coded values for existing resources
Tests should not use App.config files for settings
Tests should use the built-in PowerShell functions for generating random names when unique names are necessary - this will store names in the test recording
Tests should use Start-Sleep to pause rather than Thread.Sleep

Cmdlet Signature

Cmdlet name uses an approved PowerShell verb - use the enums for VerbsCommon, VerbsCommunication, VerbsLifecycle, VerbsOther whenever possible
Cmdlet noun name uses the AzureRm prefix for management cmdlets, and the Azure prefix for data plane cmdlets
Cmdlet specifies the OutputType attribute if any output is produced; if the cmdlet produces no output, it should implement a PassThrough parameter
If the cmdlet makes changes or has side effects, it should implement ShouldProcess and have SupportShouldProcess = true specified in the cmdlet attribute. See a discussion about correct ShouldProcess implementation here
Cmdlets should derive from AzureRmCmdlet for management cmdlets, and AzureDataCmdlet for data cmdlets
If multiple parameter sets are implemented, the cmdlet should specify a DefaultParameterSetName in its cmdlet attribute

Parameters

Cmdlets should have no more than four positional parameters
Cmdlet parameter sets should be mutually exclusive - each parameter set must have at least one mandatory parameter not in other parameter sets
Parameter types should not expose types from the management library - complex parameter types should be defined in the module
Complex parameter types are discouraged - a parameter type should be simple types as often as possible. If complex types are used, they should be shallow and easily creatable from a constructor or another cmdlet
Parameters should be explicitly marked as Mandatory or not, and should contain a HelpMessage
No parameter is of type object.
- Management cmdlets should have the following parameters and aliases:
  - ResourceGroupName with (optional) alias to ResourceGroup type string marked as [ValueFromPipelineByPropertyName]
  - Name with alias to ResourceName type string marked as [ValueFromPipelineByPropertyName]
  - Location (if appropriate) type string
  - Tag, type HashTable

Parameters and the Pipeline

Complex parameters should take values from the pipeline when possible, and certainly when they match the output type of another cmdlet
Only one parameter should use ValueFromPipeline per parameter set; parameters from different parameter sets may have this attribute, but should not be convertible
No parameter is of type object
Each management cmdlet should have a parameter set that takes ResourceGroupName and Name from the pipeline by property value
For a given resource type, it should be possible to pipe the output of Get and New cmdlets to the input of Set, Update, Remove and other action cmdlets for that resource

Missed DataLakeStore upgrade. Updates for file and folder download. Tests will be in the next iteration. Fixing the connection manager overwriting. This is important to ensure that we have enough connections available for ingress/egress scenarios, and it must be set before the client is first used. Updating the help for ADLS to include new download. Signed local package for testing. Add full path property to ADLS Items Remove private package. Official is published Adding official package from nuget. Missed one disable of tracing. Fixing ADL tests and re-running them This enables ADL tests to run during CI and gets them totally updated. fix to always copy ADL session records. Add links to AzureRM.Resources.ps1 Following the pattern from compute tests. Adding records directory info for mock server This is a requirement for the tests to be discovered when run in playback mode. Fix ADL tests and update changelog This updates the change log to reflect the changes made Fixes to datetime offset for listing jobs fix to import/export logic due to inverted parameters. Missed one of the test updates test fixes for ADLA and ADLS Add specific exclusion for ADL acls Inside of ACLs we allow ":", which breaks the playback of tests, since it encodes ":", which results in a missed match in the recorded json (which contains the colon). This fixes that so playback does not encode colons for paths that start with: /webhdfs/v1/?aclspec Fix missing resume and bad merge. update to latest version of the store package. Update to latest ADLS sdks replace local feed with signed package Update tests and fix null reference bug for ADLA Remove test package. Update ADLS Filesystem tests and SDK package There was an SDK package mismatch caught by tests. Fixing this. Update to latest ADLA SDK package. Reverting the ADLA package update. Remove app.config that was added nuget update Temporary addition of package to get signed build. Missed commit of the csproj. need a signed package.

Update to latest packages Re-add support for Get/Set permissions Re-add support for removing full ACLs Update functionality of the ACL listing and setting logic Missed a save.

This is a prep commit for a PR to the release candidate while I wait on PR approval for SDK. Unchecked changes. Updates to make the compiler happy. Compilation fix that wasn't caught by VS for some reason. Make static analysis happy.

Update Microsoft.Azure.Commands.DataLakeStore.dll-help.xml (#3) Test updates and remove local packages Fix and re-run tests for ADLS More intentional with the not yet implemented APIs remove extraneous restore.config

markcowl · 2016-09-04T01:53:28Z

...akeAnalytics/Commands.DataLakeAnalytics/Commands/NewAzureRmDataLakeAnalyticsCatalogSecret.cs

@@ -55,7 +55,7 @@ public class NewAzureDataLakeAnalyticsCatalogSecret : DataLakeAnalyticsCmdletBas

        [Parameter(ValueFromPipelineByPropertyName = true, ParameterSetName = BaseParameterSetName, Position = 3,
            Mandatory = true, HelpMessage = "The host of the database to connect to in the format 'myhost.dns.com'.")]
-        public string Host { get; set; }
+        public string DatabaseHost { get; set; }


Please add an alias to the previous parameter name to avoid a breakign change. Also, be sure to avoid any breaking changes in the properties of the output type for this cmdlet as well.

markcowl · 2016-09-04T01:58:16Z

@begoldsm One overarchign concern is breakign changes - are there places where properties are renamed in the output types that could break customer scripts?

Noted a couple of breaking changes in parameters, and please remvoe comments-out cmdlets

begoldsm · 2016-09-06T16:56:08Z

@markcowl thanks! I will remove the commented out cmdlets for now. For the breaking parameter changes I will add aliases to reduce that risk. For a couple cmdlets we have to make breaking changes (for return types and our accepted object) for ACL management due to usability concerns. I will sync with you separately to figure out if there is something we need to do to help minimize the impact of the breaking change.

Also update the help.xml to reflect reality better based on the latest from the PS Cmdlet help editor.

begoldsm · 2016-09-06T19:07:54Z

@markcowl all checks have passed with your comments addressed. Can this please be considered for the current PowerShell release?

Reverted changes to ACL behavior and added obsolete warnings and deprecation warnings for existing cmdlets. In the next release, we will make these breaking changes.

find replace is bad!

begoldsm · 2016-09-07T01:27:23Z

@markcowl all tests are passing. Can you take a look and, if there are no other concerns, get this included?

markcowl · 2016-09-07T01:29:47Z

on demand run here: http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1172/

signing run here: http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-sign/627/

markcowl · 2016-09-07T01:31:50Z

...sourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/Models/DataLakeAnalyticsClient.cs

+        public void CreateCredential(string accountName, string databaseName,
+            string credentialName, string userId, string password, string hostUri)
+        {
+            /*


@begoldsm Please remove commented code

markcowl · 2016-09-07T01:40:05Z

@begoldsm - a couple of comments and questions. Note that changing a parameter from 'ValurFromPipelineByPropertyName' -> 'ValueFromPipeline', this could introduce breaking changes for scripts that use the pipeline.

The recommendation is to use a different parameter set with a different parameter name for the new Pipeline property, if thsi is an intentional change, and deprecate the old one.

markcowl · 2016-09-07T05:07:37Z

on demand run here:http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1173/

signing run here:http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-sign/628/

LGTM once the build passes

begoldsm and others added 4 commits September 2, 2016 17:41

ADL PowerShell updates for code freeze

23d2ddf

Update to latest packages Re-add support for Get/Set permissions Re-add support for removing full ACLs Update functionality of the ACL listing and setting logic Missed a save.

Initial package updates for ADL PS

2444c9b

This is a prep commit for a PR to the release candidate while I wait on PR approval for SDK. Unchecked changes. Updates to make the compiler happy. Compilation fix that wasn't caught by VS for some reason. Make static analysis happy.

Fixes for piping and display for ACLs

af57b2e

Update Microsoft.Azure.Commands.DataLakeStore.dll-help.xml (#3) Test updates and remove local packages Fix and re-run tests for ADLS More intentional with the not yet implemented APIs remove extraneous restore.config

azurecla added the cla-not-required label Sep 3, 2016

Re-record other tests due to Op name change in http

ddadf90

markcowl reviewed Sep 4, 2016
View reviewed changes

Address comments to reduce breaking changes

50d8d57

Also update the help.xml to reflect reality better based on the latest from the PS Cmdlet help editor.

begoldsm added 4 commits September 6, 2016 16:29

Address Feedback and remove breaking changes

8cc5540

Reverted changes to ACL behavior and added obsolete warnings and deprecation warnings for existing cmdlets. In the next release, we will make these breaking changes.

Remove one extra line of blank comments.

debd087

put dashes back in comment block

3c860c9

find replace is bad!

Re-run all tests to make recordings happy.

fae3f96

markcowl reviewed Sep 7, 2016
View reviewed changes

removed commented out code per review comment.

a916c7b

begoldsm added 2 commits September 6, 2016 18:42

updates for the ACL to be both by name and the one pipeline object.

dc0c7f5

Add should process support

1284a74

markcowl merged commit 884d00a into Azure:release-2.1.0 Sep 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ADL Updates to latest Package #2892

ADL Updates to latest Package #2892

Uh oh!

begoldsm commented Sep 3, 2016 •

edited

Loading

Uh oh!

markcowl Sep 4, 2016

Uh oh!

markcowl commented Sep 4, 2016

Uh oh!

begoldsm commented Sep 6, 2016

Uh oh!

begoldsm commented Sep 6, 2016

Uh oh!

begoldsm commented Sep 7, 2016

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

markcowl Sep 7, 2016

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

Uh oh!

ADL Updates to latest Package #2892

ADL Updates to latest Package #2892

Uh oh!

Conversation

begoldsm commented Sep 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments

9. Add support for getting information about account encryption states (the ability to encrypt will come in a future release).

Overall Changes

General

Tests

Cmdlet Signature

Parameters

Parameters and the Pipeline

Uh oh!

markcowl Sep 4, 2016

Choose a reason for hiding this comment

Uh oh!

markcowl commented Sep 4, 2016

Uh oh!

begoldsm commented Sep 6, 2016

Uh oh!

begoldsm commented Sep 6, 2016

Uh oh!

begoldsm commented Sep 7, 2016

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

markcowl Sep 7, 2016

Choose a reason for hiding this comment

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

markcowl commented Sep 7, 2016

Uh oh!

Uh oh!

begoldsm commented Sep 3, 2016 •

edited

Loading