Skip to content

[ADLA - ADLS] - Object refactoring #4718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Oct 20, 2017
Merged

[ADLA - ADLS] - Object refactoring #4718

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e61ff71
Ben's object changes
ro-joowan Oct 5, 2017
efaf9f7
Revising changelog to note the Obsolete tag of the Basic objects
ro-joowan Oct 5, 2017
57c95cc
ADLA SessionRecords
ro-joowan Oct 5, 2017
f4bec0a
ADLS SessionRecords
ro-joowan Oct 5, 2017
b4ead11
Revising changelog to note the breaking change
ro-joowan Oct 9, 2017
4885905
Removing ClientRuntime, ClientRuntime.Azure, and Newtonsoft.Json
ro-joowan Oct 18, 2017
f161741
Adding breaking changes to BreakingChangeIssues.csv
ro-joowan Oct 18, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion src/ResourceManager/DataLakeAnalytics/ChangeLog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,16 @@
- Additional information about change #1
-->
## Current Release
* [BREAKING CHANGE] Removed the Obsolete Properties fields in PSDataLakeAnalyticsAccount.cs and its assoicated files
* Changed one of the two OutputTypes of Get-AzureRmDataLakeAnalyticsAccount
- List<DataLakeAnalyticsAccount> to List<PSDataLakeAnalyticsAccountBasic>
- The properties of PSDataLakeAnalyticsAccountBasic is a strict subset of the properties of DataLakeAnalyticsAccount
- The additional properties that are in DataLakeAnalyticsAccount are not returned by the service. Therefore, this change is to reflect this accurately. These additional properties are still in PSDataLakeAnalyticsAccountBasic, but they are tagged as Obsolete.
* Changed one of the two OutputTypes of Get-AzureRmDataLakeAnalyticsJob
- List<JobInformation> to List<PSJobInformationBasic>
- The properties of PSJobInformationBasic is a strict subset of the properties of JobInformation
- The additional properties that are in JobInformation are not returned by the service. Therefore, this change is to reflect this accurately. These additional properties are still in PSJobInformationBasic, but they are tagged as Obsolete.
* Updated the cmdlet logic of Submit-AzureRmDataLakeAnalyticsJob because SubmitJob and BuildJob now require an explicit parameter object rather than the generic JobInformation object

## Version 3.4.0

Expand Down Expand Up @@ -93,4 +103,4 @@
- Better error messaging and support for invalid input
* General help improvements
- Clearer help for job operations
- Fixed typos and incorrect examples
- Fixed typos and incorrect examples
11 changes: 5 additions & 6 deletions .../DataLakeAnalytics/Commands.DataLakeAnalytics.Test/Commands.DataLakeAnalytics.Test.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,11 @@
</Reference>
<Reference Include="Microsoft.Azure.Management.DataLake.Analytics, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.3.0.0\lib\net452\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.3.1.2-preview\lib\net452\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.2.2.0\lib\net452\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
<Reference Include="Microsoft.Azure.Management.DataLake.Store">
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.2.3.0-preview\lib\net452\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Microsoft.Azure.Management.ResourceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
Expand Down Expand Up @@ -104,12 +103,12 @@
</Reference>
<Reference Include="Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\..\packages\Microsoft.Rest.ClientRuntime.2.3.8\lib\net452\Microsoft.Rest.ClientRuntime.dll</HintPath>
<HintPath>..\..\..\packages\Microsoft.Rest.ClientRuntime.2.3.10\lib\net452\Microsoft.Rest.ClientRuntime.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Microsoft.Rest.ClientRuntime.Azure, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\..\..\packages\Microsoft.Rest.ClientRuntime.Azure.3.3.7\lib\net452\Microsoft.Rest.ClientRuntime.Azure.dll</HintPath>
<HintPath>..\..\..\packages\Microsoft.Rest.ClientRuntime.Azure.3.3.10\lib\net452\Microsoft.Rest.ClientRuntime.Azure.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="Microsoft.Rest.ClientRuntime.Azure.Authentication, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
Expand Down
20 changes: 16 additions & 4 deletions ...anager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaAliasTests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ function Test-DataLakeAnalyticsJobRelationships
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# submit a job
$guidForJob = [Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::GenerateGuid("relationTest01")

Expand Down Expand Up @@ -576,6 +580,10 @@ function Test-DataLakeAnalyticsJob
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# submit a job
$guidForJob = [Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::GenerateGuid("jobTest02")
[Microsoft.Azure.Commands.DataLakeAnalytics.Models.DataLakeAnalyticsClient]::JobIdQueue.Enqueue($guidForJob)
Expand All @@ -595,12 +603,12 @@ function Test-DataLakeAnalyticsJob

Assert-NotNull {Get-AdlJob -AccountName $accountName}

$jobsWithDateOffset = Get-AdlJob -AccountName $accountName -SubmittedAfter $([DateTimeOffset]($nowTime).AddMinutes(-5))
$jobsWithDateOffset = Get-AdlJob -AccountName $accountName -SubmittedAfter $([DateTimeOffset]($nowTime).AddMinutes(-10))

Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted after five miuntes ago"
Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted after ten miuntes ago"

# we add five minutes to ensure that the timing is right, since we are using the account creation time, and not truly "now"
$jobsWithDateOffset = Get-AdlJob -AccountName $accountName -SubmittedBefore $([DateTimeOffset]($nowTime).AddMinutes(5))
# we add ten minutes to ensure that the timing is right, since we are using the account creation time, and not truly "now"
$jobsWithDateOffset = Get-AdlJob -AccountName $accountName -SubmittedBefore $([DateTimeOffset]($nowTime).AddMinutes(10))

Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted before right now"

Expand Down Expand Up @@ -817,6 +825,10 @@ function Test-DataLakeAnalyticsCatalog
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(30000)
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# Run a job to create the catalog items (except secret and credential)
$scriptTemplate = @"
Expand Down
21 changes: 17 additions & 4 deletions ...urceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaTests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,10 @@ function Test-DataLakeAnalyticsJob
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# submit a job
$guidForJob = [Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::GenerateGuid("jobTest01")
[Microsoft.Azure.Commands.DataLakeAnalytics.Models.DataLakeAnalyticsClient]::JobIdQueue.Enqueue($guidForJob)
Expand All @@ -485,11 +489,12 @@ function Test-DataLakeAnalyticsJob

Assert-NotNull {Get-AzureRmDataLakeAnalyticsJob -AccountName $accountName}

$jobsWithDateOffset = Get-AzureRmDataLakeAnalyticsJob -AccountName $accountName -SubmittedAfter $([DateTimeOffset]($nowTime).AddMinutes(-5))
$jobsWithDateOffset = Get-AzureRmDataLakeAnalyticsJob -AccountName $accountName -SubmittedAfter $([DateTimeOffset]($nowTime).AddMinutes(-10))

Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted after ten miuntes ago"

Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted after five miuntes ago"
# we add five minutes to ensure that the timing is right, since we are using the account creation time, and not truly "now"
$jobsWithDateOffset = Get-AzureRmDataLakeAnalyticsJob -AccountName $accountName -SubmittedBefore $([DateTimeOffset]($nowTime).AddMinutes(5))
# we add ten minutes to ensure that the timing is right, since we are using the account creation time, and not truly "now"
$jobsWithDateOffset = Get-AzureRmDataLakeAnalyticsJob -AccountName $accountName -SubmittedBefore $([DateTimeOffset]($nowTime).AddMinutes(10))

Assert-True {$jobsWithDateOffset.Count -gt 0} "Failed to retrieve jobs submitted before right now"

Expand Down Expand Up @@ -553,6 +558,10 @@ function Test-DataLakeAnalyticsJobRelationships
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# submit a job
$guidForJob = [Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::GenerateGuid("relationTest01")

Expand Down Expand Up @@ -815,6 +824,10 @@ function Test-DataLakeAnalyticsCatalog
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(30000)
Assert-False {$i -eq 60} "dataLakeAnalytics accounts not in succeeded state even after 30 min."
}

# Wait for 5 minutes for the server to restore the account cache
# Without this, the test will pass non-deterministically
[Microsoft.Rest.ClientRuntime.Azure.TestFramework.TestUtilities]::Wait(300000)

# Run a job to create the catalog items (except secret and credential)
$scriptTemplate = @"
Expand Down
Loading