Adding DataLake commands to RM #1096

begoldsm · 2015-10-09T19:20:48Z

This change updates the Authentication package to the latest + DataLake endpoint changes.
It brings DataLake Store and DataLake Analytics cmdlets into the public repo
It uses the latest hyak.common which fixes a 500 error issue resulting in timeout.

Check in data lake store and analytics Check in changes for HDI to use data lake Update everything to use my updated version of the authentication package (which includes data lake endpoints)

For publishing support until these are published.

add correct Auth package update PSAzureEnvironment to include DataLake endpoints Add datalake to the refactoring package.

There are bugs in private HDI that need to be fixed before being ported.

Update nuget packages and fix the client due to rename miss of DataLakeAccount -> DataLakeStoreAccount

Drop download chunk max size to 32mb fix the AzureResourceManager.psd1 to properly load sub psd1 files.

azuresdkci · 2015-10-09T19:20:49Z

Can one of the admins verify this patch?

azurecla · 2015-10-09T19:20:52Z

Hi @begoldsm, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (begoldsm). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, or work for Microsoft Open Technologies, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

hovsepm · 2015-10-09T21:50:34Z

@azuresdkci add to whitelist

added "Store"

markcowl · 2015-10-12T22:39:21Z

@begoldsm Need to enable nuget package restore for the new projects

we already ensure that all paths start with a "/". this was adding another slash.

Backout visual studio package restore functionality.

this is using the new api versions

ogail · 2015-10-21T20:47:53Z

...ceManager/DataLakeStore/Commands.DataLakeStore/Models/DataLakeStoreItemPermissionInstance.cs

+                // Now do the conversion into a short
+                var parsedShort = short.Parse(convertedPermissions);
+
+                if (parsedShort < 0 || parsedShort > 777)


better defining const in vars: MinPort, MaxPort or similar

ogail · 2015-10-21T21:14:44Z

Add your tests dll to https://github.com/Azure/azure-powershell/blob/dev/AzurePowershell.Test.targets#L65 so it gets executed

ogail · 2015-10-21T21:34:11Z

src/ResourceManager/Automation/Commands.Automation.Test/Commands.Automation.Test.csproj

-      <HintPath>..\..\..\packages\Microsoft.Azure.Common.Authentication.1.3.3-preview\lib\net45\Microsoft.Azure.Common.Authentication.dll</HintPath>
-      <Private>True</Private>
+      <SpecificVersion>False</SpecificVersion>
+      <HintPath>..\..\..\packages\Microsoft.Azure.Common.Authentication.1.4.1-preview\lib\net45\Microsoft.Azure.Common.Authentication.dll</HintPath>


This package with version 1.4.1 looks like its not published on NuGet, please make sure to:

Have the change in https://github.com/Azure/azure-sdk-for-net/tree/AutoRest (this is AutoRest branch)

Publish the package from there

Update PowerShell PR to use the one published on NuGet

I will do that once my PR for AutoRest is published. The problem is that there are more changes that need to happen in PowerShell and autorest for that to work, since my package is based off version 1.3.3 and version 1.3.4 breaks the powershell. As a result, all my packages will be private until we are at a point where everything is properly published and ready to be upgraded in PowerShell.

Fixes for the uploader.

Fix bug in filtering for listing jobs Fix tests and include them for execution: NOTE: this is not a comprehensive suite, and more tests should be added here when there is time.

Fix move, join cmdlets to properly go to the right URI Optimize remove since ConfirmAction does the if/else work.

This includes the 30 second timeout for uploader operations. In the future this should be configurable, but for now if a store request takes longer than 30 seconds, it is considered a failure anyway.

This avoids conflicts when syncing from dev (hopefully).

This is the new version of the authentication SDK with data lake changes. The one that is being checked into the public repo will have this version, so the private one will just quit being used as soon as the new public one is published.

this avoids test failures due to unfound DLLs

This includes a change that makes the filesystem package testable and makes all requests ignore auto redirects for the data lake filesystem client.

markcowl · 2015-10-28T00:19:50Z

@begoldsm As we discussed, please remove localFeed entities

updated hdi sdk version and fixed build failures

removing the local feed since all packages are now live.

hovsepm · 2015-10-28T20:53:05Z

@begoldsm please fix test failures in on-demand job run.

begoldsm · 2015-10-28T21:09:21Z

@markcowl @hovsepm the on demand run is failing due to issues that are not mine. As you can see from the job history, Mark disabled these sql tests before to get them to pass (see job 249 vs job 250). @markcowl, can you please investigate why these sql tests are running again as part of this?

markcowl · 2015-10-28T21:15:37Z

@begoldsm you most likely need to pull the latest from the dev branch for these to pass

markcowl · 2015-10-28T21:47:07Z

@begoldsm you will need to pull the latest. The fix for this was pushed a couple of hours ago.

markcowl · 2015-10-28T22:45:01Z

@azuresdkci retest this please

hovsepm · 2015-10-29T00:21:39Z

on-demand job http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-on-demand/277/

Adding DataLake commands to RM

begoldsm added 14 commits October 7, 2015 14:57

ignore all ghost doc files

3648b05

Check in DataLake changes

94a61fd

Check in data lake store and analytics Check in changes for HDI to use data lake Update everything to use my updated version of the authentication package (which includes data lake endpoints)

Adding local feed packages

6f87811

For publishing support until these are published.

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

483d0a6

Fixes for DataLake

26dee88

add correct Auth package update PSAzureEnvironment to include DataLake endpoints Add datalake to the refactoring package.

Fix to re-install HDI package.

771d97c

fixing HDI reference... again.

7d7d6f4

Revert HDI to what is in public already

33f7f43

There are bugs in private HDI that need to be fixed before being ported.

Add the "Rm" in the name of cmdlets

cfeaf67

Update wix file

8d44ffe

Update nuget packages and fix the client due to rename miss of DataLakeAccount -> DataLakeStoreAccount

Update AzureRM and guids

060566a

Upping my version

2067010

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

ec15bbd

fix the large msi lower download size chunks

de015fa

Drop download chunk max size to 32mb fix the AzureResourceManager.psd1 to properly load sub psd1 files.

azurecla added the cla-not-required label Oct 9, 2015

begoldsm added 2 commits October 9, 2015 15:53

Missed file rename

5270274

added "Store"

bug fix to ensure updates work

cd26543

begoldsm added 9 commits October 12, 2015 19:07

updates for new namespace

081ad04

Adding nuget restore to the new DataLake projects

0657d17

missed an update.

a514aac

fixing bug with concat

d5575c5

we already ensure that all paths start with a "/". this was adding another slash.

Incomplete fix for concat

607bb2a

Update nuget packages.

9958461

Backout visual studio package restore functionality.

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

be7a4da

rev the filesystem package

ddf7ade

this is using the new api versions

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

9d11872

ogail reviewed Oct 21, 2015
View reviewed changes

begoldsm added 9 commits October 21, 2015 21:18

Address Ogail's comments

0d93cb9

Fixes for the uploader.

Address Ogail Comments, bug fix

765616a

Fix bug in filtering for listing jobs Fix tests and include them for execution: NOTE: this is not a comprehensive suite, and more tests should be added here when there is time.

Bug fix for store cmdlets

d379b37

Fix move, join cmdlets to properly go to the right URI Optimize remove since ConfirmAction does the if/else work.

Update packages for uploader

b6f9c7f

This includes the 30 second timeout for uploader operations. In the future this should be configurable, but for now if a store request takes longer than 30 seconds, it is considered a failure anyway.

"Update all auth packages"

03923b4

This avoids conflicts when syncing from dev (hopefully).

Sync to latest dev changes.

5c1f669

update to use latest version of the auth sdk

55b0fde

This is the new version of the authentication SDK with data lake changes. The one that is being checked into the public repo will have this version, so the private one will just quit being used as soon as the new public one is published.

Fix all the tests to use the new Test framework

1e8e4b2

this avoids test failures due to unfound DLLs

Update the filesystem package.

ff61ac8

This includes a change that makes the filesystem package testable and makes all requests ignore auto redirects for the data lake filesystem client.

yoreddy and others added 4 commits October 28, 2015 13:19

updated hdi sdk version and fixed build failures

893a226

Merge pull request #1 from yoreddy/dev

ce28783

updated hdi sdk version and fixed build failures

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

cba5a0f

remove all local feed packages

f3bbb7d

removing the local feed since all packages are now live.

Merge branch 'dev' of https://github.com/Azure/azure-powershell into dev

563d135

hovsepm pushed a commit that referenced this pull request Oct 29, 2015

Merge pull request #1096 from begoldsm/dev

01f2ffe

Adding DataLake commands to RM

hovsepm merged commit 01f2ffe into Azure:dev Oct 29, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding DataLake commands to RM #1096

Adding DataLake commands to RM #1096

Uh oh!

begoldsm commented Oct 9, 2015

Uh oh!

azuresdkci commented Oct 9, 2015

Uh oh!

azurecla commented Oct 9, 2015

Uh oh!

hovsepm commented Oct 9, 2015

Uh oh!

markcowl commented Oct 12, 2015

Uh oh!

ogail Oct 21, 2015

Uh oh!

ogail commented Oct 21, 2015

Uh oh!

ogail Oct 21, 2015

Uh oh!

begoldsm Oct 21, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

hovsepm commented Oct 28, 2015

Uh oh!

begoldsm commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

hovsepm commented Oct 29, 2015

Uh oh!

Uh oh!

Adding DataLake commands to RM #1096

Adding DataLake commands to RM #1096

Uh oh!

Conversation

begoldsm commented Oct 9, 2015

Uh oh!

azuresdkci commented Oct 9, 2015

Uh oh!

azurecla commented Oct 9, 2015

Uh oh!

hovsepm commented Oct 9, 2015

Uh oh!

markcowl commented Oct 12, 2015

Uh oh!

ogail Oct 21, 2015

Choose a reason for hiding this comment

Uh oh!

ogail commented Oct 21, 2015

Uh oh!

ogail Oct 21, 2015

Choose a reason for hiding this comment

Uh oh!

begoldsm Oct 21, 2015

Choose a reason for hiding this comment

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

hovsepm commented Oct 28, 2015

Uh oh!

begoldsm commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

markcowl commented Oct 28, 2015

Uh oh!

hovsepm commented Oct 29, 2015

Uh oh!

Uh oh!