Skip to content

Commit acaa012

Browse files
begoldsmcormacpayne
begoldsm
authored and
cormacpayne
committed
Update to latest ADL Packages (Azure#3477)
* Update PowerShell to latest package for ADL This includes a rework of the PowerShell friendly client and smart settings for file and folder upload/download. I also re-ran all tests to ensure that they are all still passing, which resulted in discovering that the MASK is now properly populated immediately when creating an account, so had to update the permissions tests. * Add support for returning a set number of jobs Also remove test package. * Add warning for disabled IDP and firewall states Add warning to deletes. Fix from using private package Was using private package previously that hadn't fully refactored Upload -> Transfer Missed removal of Uploader package in tests. Fix the wxi file. * Update test recordings. * Removing strict validation from location param This is difficult to maintain and the RP itself should handle this when the attempt is sent. Unfortunately this has the by-product of removing tab completion from the cmdlet for locations, but maintaining this set when new regions are on boarded does not scale. * Add Firewall Support, Rerun tests, update changelog * Fixing code bug and broken tests. * Need to add json to csproj. * Address PR comments about help
1 parent 7bb65be commit acaa012

File tree

83 files changed

+20202
-32930
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

83 files changed

+20202
-32930
lines changed

‎setup/azurecmdfiles.wxi

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -981,9 +981,6 @@
981981
<Component Id="cmp3C96010EFA751762A76B66B811613835" Guid="*">
982982
<File Id="fil509D1E7494560B2BB3462B8FAD7954CF" KeyPath="yes" Source="$(var.sourceDir)\ResourceManager\AzureResourceManager\AzureRM.DataLakeStore\Microsoft.Azure.Management.DataLake.Store.dll" />
983983
</Component>
984-
<Component Id="cmpFECE1A517C9182672B042869C46847FC" Guid="*">
985-
<File Id="fil00C18C352C99B9BCFA42EFFACE0AEFF7" KeyPath="yes" Source="$(var.sourceDir)\ResourceManager\AzureResourceManager\AzureRM.DataLakeStore\Microsoft.Azure.Management.DataLake.StoreUploader.dll" />
986-
</Component>
987984
<Component Id="cmp500DAF931B66664E6A3308FAC825093C" Guid="*">
988985
<File Id="filB384FAE2600A5CAE72336384DBA3B581" KeyPath="yes" Source="$(var.sourceDir)\ResourceManager\AzureResourceManager\AzureRM.DataLakeStore\Microsoft.Azure.Management.ResourceManager.dll" />
989986
</Component>
@@ -6038,7 +6035,6 @@
60386035
<ComponentRef Id="cmp87FB43928F7A2ECF62C0141446D21A9F" />
60396036
<ComponentRef Id="cmpD2292657EF3D9CEAC5D4FA7DEB3E7E8E" />
60406037
<ComponentRef Id="cmp3C96010EFA751762A76B66B811613835" />
6041-
<ComponentRef Id="cmpFECE1A517C9182672B042869C46847FC" />
60426038
<ComponentRef Id="cmp500DAF931B66664E6A3308FAC825093C" />
60436039
<ComponentRef Id="cmpFCFE1AC675F0BA7336B392A419D70042" />
60446040
<ComponentRef Id="cmpB41201E194E7639BC9C33ADECC894426" />

‎src/ResourceManager/DataLakeAnalytics/AzureRM.DataLakeAnalytics.psd1

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,11 @@ CmdletsToExport = 'Get-AzureRmDataLakeAnalyticsDataSource',
9292
'Submit-AzureRmDataLakeAnalyticsJob',
9393
'New-AzureRmDataLakeAnalyticsAccount',
9494
'Remove-AzureRmDataLakeAnalyticsAccount',
95-
'Set-AzureRmDataLakeAnalyticsAccount'
95+
'Set-AzureRmDataLakeAnalyticsAccount',
96+
'Add-AzureRmDataLakeAnalyticsFirewallRule',
97+
'Get-AzureRmDataLakeAnalyticsFirewallRule',
98+
'Set-AzureRmDataLakeAnalyticsFirewallRule',
99+
'Remove-AzureRmDataLakeAnalyticsFirewallRule'
96100

97101
# Variables to export from this module
98102
# VariablesToExport = @()
@@ -106,7 +110,9 @@ AliasesToExport = 'Get-AdlAnalyticsDataSource', 'New-AdlCatalogCredential',
106110
'Test-AdlAnalyticsAccount', 'Remove-AdlAnalyticsDataSource',
107111
'Add-AdlAnalyticsDataSource', 'Stop-AdlJob', 'Get-AdlJob',
108112
'Get-AdlAnalyticsAccount', 'Submit-AdlJob', 'New-AdlAnalyticsAccount',
109-
'Remove-AdlAnalyticsAccount', 'Set-AdlAnalyticsAccount'
113+
'Remove-AdlAnalyticsAccount', 'Set-AdlAnalyticsAccount',
114+
'Add-AdlAnalyticsFirewallRule', 'Get-AdlAnalyticsFirewallRule',
115+
'Set-AdlAnalyticsFirewallRule', 'Remove-AdlAnalyticsFirewallRule'
110116

111117
# DSC resources to export from this module
112118
# DscResourcesToExport = @()

‎src/ResourceManager/DataLakeAnalytics/ChangeLog.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,16 @@
1818
- Additional information about change #1
1919
-->
2020
## Current Release
21+
* Add Firewall Rule support to Data Lake Analytics:
22+
- Add-AzureRMDataLakeAnalyticsFirewallRule
23+
- Get-AzureRMDataLakeAnalyticsFirewallRule
24+
- Set-AzureRMDataLakeAnalyticsFirewallRule
25+
- Remove-AzureRMDataLakeAnalyticsFirewallRule
26+
- Set-AzureRMDataLakeAnalyticsAccount supports enabling/disabling the firewall and allowing/blocking Azure originating IPs through the firewall
27+
- Warnings will be raised if updating firewall rules when the firewall is disabled
28+
* Fix Get-AzureRMDataLakeAnalyticsJob functionality:
29+
- Top now correctly returns the number of jobs specified. The default number of jobs to return is 500. The more jobs requested the longer the command will take.
30+
* Remove explicit restrictions on resource locations. If Data Lake Analytics is not supported in a region, we will surface an error from the service.
2131

2232
## Version 2.5.0
2333
* Update Get-AdlJob to support Top parameter

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/Commands.DataLakeAnalytics.Test.csproj

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,12 @@
6464
</Reference>
6565
<Reference Include="Microsoft.Azure.Management.DataLake.Analytics, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
6666
<SpecificVersion>False</SpecificVersion>
67-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.2.0.0\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
67+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.2.0.1-preview\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
6868
<Private>True</Private>
6969
</Reference>
70-
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
70+
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
7171
<SpecificVersion>False</SpecificVersion>
72-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.1.0.3\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
72+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.2.0.1-preview\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
7373
<Private>True</Private>
7474
</Reference>
7575
<Reference Include="Microsoft.Azure.Management.ResourceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
@@ -230,6 +230,9 @@
230230
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaAliasTests\TestAdlaCatalog.json">
231231
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
232232
</None>
233+
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaAliasTests\TestAdlaFirewallRules.json">
234+
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
235+
</None>
233236
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaAliasTests\TestAdlaJob.json">
234237
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
235238
</None>
@@ -248,6 +251,9 @@
248251
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests\TestAdlaCatalog.json">
249252
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
250253
</None>
254+
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests\TestAdlaFirewallRules.json">
255+
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
256+
</None>
251257
<None Include="SessionRecords\Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests\TestAdlaJob.json">
252258
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
253259
</None>

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaAliasTests.cs

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,12 +38,22 @@ public void TestAdlaAccount()
3838
[Trait(Category.AcceptanceType, Category.CheckIn)]
3939
public void TestAdlaAccountTiers()
4040
{
41-
AdlaTestsBase.NewInstance.RunPsTest(true,
41+
AdlaTestsBase.NewInstance.RunPsTest(false,
4242
string.Format(
4343
"Test-DataLakeAnalyticsAccountTiers -location '{0}'",
4444
AdlaTestsBase.resourceGroupLocation));
4545
}
4646

47+
[Fact]
48+
[Trait(Category.AcceptanceType, Category.CheckIn)]
49+
public void TestAdlaFirewallRules()
50+
{
51+
AdlaTestsBase.NewInstance.RunPsTest(false,
52+
string.Format(
53+
"Test-DataLakeAnalyticsFirewall -location '{0}'",
54+
AdlaTestsBase.resourceGroupLocation));
55+
}
56+
4757
[Fact]
4858
[Trait(Category.AcceptanceType, Category.CheckIn)]
4959
public void TestAdlaCatalog()

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaAliasTests.ps1

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,95 @@
11
<#
22
.SYNOPSIS
3+
Tests DataLakeAnalytics Account firewall rule lifecycle (Create, Update, Get, List, Delete).
4+
#>
5+
function Test-DataLakeAnalyticsFirewall
6+
{
7+
param
8+
(
9+
$resourceGroupName = (Get-ResourceGroupName),
10+
$accountName = (Get-DataLakeAnalyticsAccountName),
11+
$dataLakeAccountName = (Get-DataLakeStoreAccountName),
12+
$location = "West US"
13+
)
14+
15+
try
16+
{
17+
# Creating Account
18+
New-AzureRmResourceGroup -Name $resourceGroupName -Location $location
19+
20+
# Test to make sure the account doesn't exist
21+
Assert-False {Test-AdlAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName}
22+
# Test it without specifying a resource group
23+
Assert-False {Test-AdlAnalyticsAccount -Name $accountName}
24+
25+
New-AdlStore -ResourceGroupName $resourceGroupName -Name $dataLakeAccountName -Location $location
26+
27+
$accountCreated = New-AdlAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName -Location $location -DefaultDataLakeStore $dataLakeAccountName
28+
29+
Assert-AreEqual $accountName $accountCreated.Name
30+
Assert-AreEqual $location $accountCreated.Location
31+
Assert-AreEqual "Microsoft.DataLakeAnalytics/accounts" $accountCreated.Type
32+
Assert-True {$accountCreated.Id -like "*$resourceGroupName*"}
33+
34+
# In loop to check if account exists
35+
for ($i = 0; $i -le 60; $i++)
36+
{
37+
[array]$accountGet = Get-AdlAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName
38+
if ($accountGet[0].ProvisioningState -like "Succeeded")
39+
{
40+
Assert-AreEqual $accountName $accountGet[0].Name
41+
Assert-AreEqual $location $accountGet[0].Location
42+
Assert-AreEqual "Microsoft.DataLakeAnalytics/accounts" $accountGet[0].Type
43+
Assert-True {$accountGet[0].Id -like "*$resourceGroupName*"}
44+
break
45+
}
46+
47+
Write-Host "account not yet provisioned. current state: $($accountGet[0].ProvisioningState)"
48+
[Microsoft.WindowsAzure.Commands.Utilities.Common.TestMockSupport]::Delay(30000)
49+
Assert-False {$i -eq 60} " Data Lake Analytics account is not in succeeded state even after 30 min."
50+
}
51+
52+
# Test to make sure the account does exist
53+
Assert-True {Test-AdlAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName}
54+
55+
# Test to enable the firewall as well as allowing azure IPs
56+
Assert-AreEqual "Disabled" $accountCreated.FirewallState
57+
Assert-AreEqual "Disabled" $accountCreated.FirewallAllowAzureIps
58+
59+
$accountSet = Set-AdlAnalyticsAccount -Name $accountName -FirewallState "Enabled" -AllowAzureIpState "Enabled"
60+
61+
Assert-AreEqual "Enabled" $accountSet.FirewallState
62+
Assert-AreEqual "Enabled" $accountSet.FirewallAllowAzureIps
63+
64+
$firewallRuleName = getAssetName
65+
$startIp = "127.0.0.1"
66+
$endIp = "127.0.0.2"
67+
# Add a firewall rule
68+
Add-AdlAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName -StartIpAddress $startIp -EndIpAddress $endIp
69+
70+
# Get the firewall rule
71+
$result = Get-AdlAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName
72+
Assert-AreEqual $firewallRuleName $result.Name
73+
Assert-AreEqual $startIp $result.StartIpAddress
74+
Assert-AreEqual $endIp $result.EndIpAddress
75+
76+
# remove the firewall rule
77+
Remove-AdlAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName
78+
79+
# Make sure get throws.
80+
Assert-Throws {Get-AdlAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName}
81+
}
82+
finally
83+
{
84+
# cleanup the resource group that was used in case it still exists. This is a best effort task, we ignore failures here.
85+
Invoke-HandledCmdlet -Command {Remove-AdlAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
86+
Invoke-HandledCmdlet -Command {Remove-AdlStore -ResourceGroupName $resourceGroupName -Name $dataLakeAccountName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
87+
Invoke-HandledCmdlet -Command {Remove-AzureRmResourceGroup -Name $resourceGroupName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
88+
}
89+
}
90+
91+
<#
92+
.SYNOPSIS
393
Tests DataLakeAnalytics Account Lifecycle (Create, Update, Get, List, Delete).
494
#>
595
function Test-DataLakeAnalyticsAccount

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaTests.cs

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,12 +38,22 @@ public void TestAdlaAccount()
3838
[Trait(Category.AcceptanceType, Category.CheckIn)]
3939
public void TestAdlaAccountTiers()
4040
{
41-
AdlaTestsBase.NewInstance.RunPsTest(true,
41+
AdlaTestsBase.NewInstance.RunPsTest(false,
4242
string.Format(
4343
"Test-DataLakeAnalyticsAccountTiers -location '{0}'",
4444
AdlaTestsBase.resourceGroupLocation));
4545
}
4646

47+
[Fact]
48+
[Trait(Category.AcceptanceType, Category.CheckIn)]
49+
public void TestAdlaFirewallRules()
50+
{
51+
AdlaTestsBase.NewInstance.RunPsTest(false,
52+
string.Format(
53+
"Test-DataLakeAnalyticsFirewall -location '{0}'",
54+
AdlaTestsBase.resourceGroupLocation));
55+
}
56+
4757
[Fact]
4858
[Trait(Category.AcceptanceType, Category.CheckIn)]
4959
public void TestAdlaCatalog()

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaTests.ps1

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,95 @@
11
<#
22
.SYNOPSIS
3+
Tests DataLakeAnalytics Account firewall rule lifecycle (Create, Update, Get, List, Delete).
4+
#>
5+
function Test-DataLakeAnalyticsFirewall
6+
{
7+
param
8+
(
9+
$resourceGroupName = (Get-ResourceGroupName),
10+
$accountName = (Get-DataLakeAnalyticsAccountName),
11+
$dataLakeAccountName = (Get-DataLakeStoreAccountName),
12+
$location = "West US"
13+
)
14+
15+
try
16+
{
17+
# Creating Account
18+
New-AzureRmResourceGroup -Name $resourceGroupName -Location $location
19+
20+
# Test to make sure the account doesn't exist
21+
Assert-False {Test-AzureRMDataLakeAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName}
22+
# Test it without specifying a resource group
23+
Assert-False {Test-AzureRMDataLakeAnalyticsAccount -Name $accountName}
24+
25+
New-AzureRMDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeAccountName -Location $location
26+
27+
$accountCreated = New-AzureRMDataLakeAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName -Location $location -DefaultDataLakeStore $dataLakeAccountName
28+
29+
Assert-AreEqual $accountName $accountCreated.Name
30+
Assert-AreEqual $location $accountCreated.Location
31+
Assert-AreEqual "Microsoft.DataLakeAnalytics/accounts" $accountCreated.Type
32+
Assert-True {$accountCreated.Id -like "*$resourceGroupName*"}
33+
34+
# In loop to check if account exists
35+
for ($i = 0; $i -le 60; $i++)
36+
{
37+
[array]$accountGet = Get-AzureRMDataLakeAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName
38+
if ($accountGet[0].ProvisioningState -like "Succeeded")
39+
{
40+
Assert-AreEqual $accountName $accountGet[0].Name
41+
Assert-AreEqual $location $accountGet[0].Location
42+
Assert-AreEqual "Microsoft.DataLakeAnalytics/accounts" $accountGet[0].Type
43+
Assert-True {$accountGet[0].Id -like "*$resourceGroupName*"}
44+
break
45+
}
46+
47+
Write-Host "account not yet provisioned. current state: $($accountGet[0].ProvisioningState)"
48+
[Microsoft.WindowsAzure.Commands.Utilities.Common.TestMockSupport]::Delay(30000)
49+
Assert-False {$i -eq 60} " Data Lake Analytics account is not in succeeded state even after 30 min."
50+
}
51+
52+
# Test to make sure the account does exist
53+
Assert-True {Test-AzureRMDataLakeAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName}
54+
55+
# Test to enable the firewall as well as allowing azure IPs
56+
Assert-AreEqual "Disabled" $accountCreated.FirewallState
57+
Assert-AreEqual "Disabled" $accountCreated.FirewallAllowAzureIps
58+
59+
$accountSet = Set-AzureRMDataLakeAnalyticsAccount -Name $accountName -FirewallState "Enabled" -AllowAzureIpState "Enabled"
60+
61+
Assert-AreEqual "Enabled" $accountSet.FirewallState
62+
Assert-AreEqual "Enabled" $accountSet.FirewallAllowAzureIps
63+
64+
$firewallRuleName = getAssetName
65+
$startIp = "127.0.0.1"
66+
$endIp = "127.0.0.2"
67+
# Add a firewall rule
68+
Add-AzureRMDataLakeAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName -StartIpAddress $startIp -EndIpAddress $endIp
69+
70+
# Get the firewall rule
71+
$result = Get-AzureRMDataLakeAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName
72+
Assert-AreEqual $firewallRuleName $result.Name
73+
Assert-AreEqual $startIp $result.StartIpAddress
74+
Assert-AreEqual $endIp $result.EndIpAddress
75+
76+
# remove the firewall rule
77+
Remove-AzureRMDataLakeAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName
78+
79+
# Make sure get throws.
80+
Assert-Throws {Get-AzureRMDataLakeAnalyticsFirewallRule -AccountName $accountName -Name $firewallRuleName}
81+
}
82+
finally
83+
{
84+
# cleanup the resource group that was used in case it still exists. This is a best effort task, we ignore failures here.
85+
Invoke-HandledCmdlet -Command {Remove-AzureRMDataLakeAnalyticsAccount -ResourceGroupName $resourceGroupName -Name $accountName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
86+
Invoke-HandledCmdlet -Command {Remove-AzureRMDataLakeStoreAccount -ResourceGroupName $resourceGroupName -Name $dataLakeAccountName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
87+
Invoke-HandledCmdlet -Command {Remove-AzureRmResourceGroup -Name $resourceGroupName -Force -ErrorAction SilentlyContinue} -IgnoreFailures
88+
}
89+
}
90+
91+
<#
92+
.SYNOPSIS
393
Tests DataLakeAnalytics Account Lifecycle (Create, Update, Get, List, Delete).
494
#>
595
function Test-DataLakeAnalyticsAccount

0 commit comments

Comments
 (0)