Skip to content

Commit 1f4b786

Browse files
author
begoldsm
committed
Update PowerShell to latest SDK
This includes the following: 1. Progress tracking and resume functionality for folder upload 2. Partition functionality added to get catalog item 3. Better error handling and exceptions for the store. 4. More robust client side timeout for long running filesystem operations. Also re-run all tests to confirm they are passing and add new tests for partition.
1 parent a822d1f commit 1f4b786

File tree

23 files changed

+3672
-27310
lines changed

23 files changed

+3672
-27310
lines changed

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/Commands.DataLakeAnalytics.Test.csproj

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,12 @@
6464
</Reference>
6565
<Reference Include="Microsoft.Azure.Management.DataLake.Analytics, Version=0.11.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
6666
<SpecificVersion>False</SpecificVersion>
67-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.0.11.2-preview\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
67+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.0.11.3-preview\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
6868
<Private>True</Private>
6969
</Reference>
70-
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=0.11.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
70+
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=0.12.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
7171
<SpecificVersion>False</SpecificVersion>
72-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.0.11.0-preview\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
72+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.0.12.0-preview\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
7373
<Private>True</Private>
7474
</Reference>
7575
<Reference Include="Microsoft.Azure.Management.Storage">

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/ScenarioTests/AdlaTests.ps1

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -470,8 +470,9 @@ function Test-DataLakeAnalyticsCatalog
470470
ClickedUrls string,
471471
INDEX idx1 //Name of index
472472
CLUSTERED (Region ASC) //Column to cluster by
473-
PARTITIONED BY HASH (Region) //Column to partition by
473+
PARTITIONED BY BUCKETS (UserId) HASH (Region) //Column to partition by
474474
);
475+
ALTER TABLE {0}.dbo.{1} ADD IF NOT EXISTS PARTITION (1);
475476
DROP FUNCTION IF EXISTS {0}.dbo.{2};
476477
477478
//create table weblogs on space-delimited website log data
@@ -598,6 +599,20 @@ function Test-DataLakeAnalyticsCatalog
598599
Assert-NotNull $specificItem "Could not retrieve the table by name"
599600
Assert-AreEqual $tableName $specificItem.Name
600601

602+
# retrieve the list of table partitions
603+
$itemList = Get-AzureRMDataLakeAnalyticsCatalogItem -AccountName $accountName -ItemType TablePartition -Path "$databaseName.dbo.$tableName"
604+
605+
Assert-NotNull $itemList "The table partition list is null"
606+
607+
Assert-True {$itemList.count -gt 0} "The table partition list is empty"
608+
609+
$itemToFind = $itemList[0]
610+
611+
# retrieve the specific table partition
612+
$specificItem = Get-AzureRMDataLakeAnalyticsCatalogItem -AccountName $accountName -ItemType TablePartition -Path "$databaseName.dbo.$tableName.$($itemToFind.Name)"
613+
Assert-NotNull $specificItem "Could not retrieve the table partition by name"
614+
Assert-AreEqual $itemToFind.Name $specificItem.Name
615+
601616
# retrieve the list of table valued functions and ensure the created tvf is in it
602617
$itemList = Get-AzureRMDataLakeAnalyticsCatalogItem -AccountName $accountName -ItemType TableValuedFunction -Path "$databaseName.dbo"
603618

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/SessionRecords/Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests/TestAdlaAccount.json

Lines changed: 716 additions & 839 deletions
Large diffs are not rendered by default.

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/SessionRecords/Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests/TestAdlaCatalog.json

Lines changed: 719 additions & 24185 deletions
Large diffs are not rendered by default.

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/SessionRecords/Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests/TestAdlaJob.json

Lines changed: 271 additions & 271 deletions
Large diffs are not rendered by default.

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/SessionRecords/Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests/TestNegativeAdlaAccount.json

Lines changed: 343 additions & 283 deletions
Large diffs are not rendered by default.

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/SessionRecords/Microsoft.Azure.Commands.DataLakeAnalytics.Test.ScenarioTests.AdlaTests/TestNegativeAdlaJob.json

Lines changed: 294 additions & 240 deletions
Large diffs are not rendered by default.

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics.Test/packages.config

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
<package id="Microsoft.Azure.Common.Dependencies" version="1.0.0" targetFramework="net45" />
66
<package id="Microsoft.Azure.Gallery" version="2.6.2-preview" targetFramework="net45" />
77
<package id="Microsoft.Azure.Management.Authorization" version="2.0.0" targetFramework="net45" />
8-
<package id="Microsoft.Azure.Management.DataLake.Analytics" version="0.11.2-preview" targetFramework="net45" />
9-
<package id="Microsoft.Azure.Management.DataLake.Store" version="0.11.0-preview" targetFramework="net45" />
8+
<package id="Microsoft.Azure.Management.DataLake.Analytics" version="0.11.3-preview" targetFramework="net45" />
9+
<package id="Microsoft.Azure.Management.DataLake.Store" version="0.12.0-preview" targetFramework="net45" />
1010
<package id="Microsoft.Azure.Management.Resources" version="2.20.0-preview" targetFramework="net45" />
1111
<package id="Microsoft.Azure.Management.Storage" version="3.0.0" targetFramework="net45" />
1212
<package id="Microsoft.Azure.Test.Framework" version="1.0.5945.28173-prerelease" targetFramework="net45" />

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/Commands.DataLakeAnalytics.csproj

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@
4747
</Reference>
4848
<Reference Include="Microsoft.Azure.Management.DataLake.Analytics, Version=0.11.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
4949
<SpecificVersion>False</SpecificVersion>
50-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.0.11.2-preview\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
50+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Analytics.0.11.3-preview\lib\net45\Microsoft.Azure.Management.DataLake.Analytics.dll</HintPath>
5151
<Private>True</Private>
5252
</Reference>
5353
<Reference Include="Microsoft.Azure.ResourceManager, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/Models/CatalogPathInstance.cs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ public class CatalogPathInstance
2727
public string DatabaseName { get; set; }
2828
public string SchemaAssemblyOrExternalDataSourceName { get; set; }
2929
public string TableOrTableValuedFunctionName { get; set; }
30-
public string TableStatisticsName { get; set; }
30+
public string TableStatisticsOrPartitionName { get; set; }
3131
public string FullCatalogItemPath { get; set; }
3232

3333
public static CatalogPathInstance Parse(string path)
@@ -85,7 +85,7 @@ public static CatalogPathInstance Parse(string path)
8585
DatabaseName = firstPart,
8686
SchemaAssemblyOrExternalDataSourceName = secondPart,
8787
TableOrTableValuedFunctionName = thirdPart,
88-
TableStatisticsName = fourthPart,
88+
TableStatisticsOrPartitionName = fourthPart,
8989
FullCatalogItemPath = path
9090
};
9191
}

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/Models/DataLakeAnalyticsClient.cs

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,19 @@ public IList<CatalogItem> GetCatalogItem(string accountName, CatalogPathInstance
498498
path.SchemaAssemblyOrExternalDataSourceName, path.TableOrTableValuedFunctionName));
499499
}
500500

501+
break;
502+
case DataLakeAnalyticsEnums.CatalogItemType.TablePartition:
503+
if (isList)
504+
{
505+
toReturn.AddRange(GetTablePartitions(accountName, path.DatabaseName,
506+
path.SchemaAssemblyOrExternalDataSourceName, path.TableOrTableValuedFunctionName));
507+
}
508+
else
509+
{
510+
toReturn.Add(GetTablePartition(accountName, path.DatabaseName,
511+
path.SchemaAssemblyOrExternalDataSourceName, path.TableOrTableValuedFunctionName, path.TableStatisticsOrPartitionName));
512+
}
513+
501514
break;
502515
case DataLakeAnalyticsEnums.CatalogItemType.TableValuedFunction:
503516
if (isList)
@@ -522,7 +535,7 @@ public IList<CatalogItem> GetCatalogItem(string accountName, CatalogPathInstance
522535
{
523536
toReturn.Add(GetTableStatistic(accountName, path.DatabaseName,
524537
path.SchemaAssemblyOrExternalDataSourceName, path.TableOrTableValuedFunctionName,
525-
path.TableStatisticsName));
538+
path.TableStatisticsOrPartitionName));
526539
}
527540

528541
break;
@@ -710,6 +723,28 @@ private IList<USqlTable> GetTables(string accountName, string databaseName,
710723
return toReturn;
711724
}
712725

726+
private USqlTablePartition GetTablePartition(string accountName, string databaseName, string schemaName,
727+
string tableName, string partitionName)
728+
{
729+
return
730+
_catalogClient.Catalog.GetTablePartition(accountName, databaseName, schemaName, tableName, partitionName);
731+
}
732+
733+
private IList<USqlTablePartition> GetTablePartitions(string accountName, string databaseName,
734+
string schemaName, string tableName)
735+
{
736+
List<USqlTablePartition> toReturn = new List<USqlTablePartition>();
737+
var response = _catalogClient.Catalog.ListTablePartitions(accountName, databaseName, schemaName, tableName);
738+
toReturn.AddRange(response);
739+
while (!string.IsNullOrEmpty(response.NextPageLink))
740+
{
741+
response = _catalogClient.Catalog.ListTablePartitionsNext(response.NextPageLink);
742+
toReturn.AddRange(response);
743+
}
744+
745+
return toReturn;
746+
}
747+
713748
private USqlTableValuedFunction GetTableValuedFunction(string accountName,
714749
string databaseName, string schemaName, string tableValuedFunctionName)
715750
{
@@ -958,6 +993,7 @@ private bool IsCatalogItemOrList(CatalogPathInstance path, DataLakeAnalyticsEnum
958993

959994
break;
960995
case DataLakeAnalyticsEnums.CatalogItemType.TableStatistics:
996+
case DataLakeAnalyticsEnums.CatalogItemType.TablePartition:
961997
if (string.IsNullOrEmpty(path.DatabaseName) ||
962998
string.IsNullOrEmpty(path.SchemaAssemblyOrExternalDataSourceName) ||
963999
string.IsNullOrEmpty(path.TableOrTableValuedFunctionName))
@@ -966,7 +1002,7 @@ private bool IsCatalogItemOrList(CatalogPathInstance path, DataLakeAnalyticsEnum
9661002
path.FullCatalogItemPath));
9671003
}
9681004

969-
if (string.IsNullOrEmpty(path.TableStatisticsName))
1005+
if (string.IsNullOrEmpty(path.TableStatisticsOrPartitionName))
9701006
{
9711007
isList = true;
9721008
}

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/Models/DataLakeAnalyticsEnums.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ public enum CatalogItemType
2222
Schema,
2323
Assembly,
2424
Table,
25+
TablePartition,
2526
TableValuedFunction,
2627
TableStatistics,
2728
ExternalDataSource,

‎src/ResourceManager/DataLakeAnalytics/Commands.DataLakeAnalytics/packages.config

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<package id="Hyak.Common" version="1.0.3" targetFramework="net45" />
44
<package id="Microsoft.Azure.Common" version="2.1.0" targetFramework="net45" />
55
<package id="Microsoft.Azure.Common.Dependencies" version="1.0.0" targetFramework="net45" />
6-
<package id="Microsoft.Azure.Management.DataLake.Analytics" version="0.11.2-preview" targetFramework="net45" />
6+
<package id="Microsoft.Azure.Management.DataLake.Analytics" version="0.11.3-preview" targetFramework="net45" />
77
<package id="Microsoft.Azure.Management.Resources" version="2.20.0-preview" targetFramework="net45" />
88
<package id="Microsoft.Bcl" version="1.1.9" targetFramework="net45" />
99
<package id="Microsoft.Bcl.Async" version="1.0.168" targetFramework="net45" />

‎src/ResourceManager/DataLakeStore/Commands.DataLakeStore.Test/Commands.DataLakeStore.Test.csproj

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,14 @@
6262
<HintPath>..\..\..\packages\Microsoft.Azure.Management.Authorization.2.0.0\lib\net40\Microsoft.Azure.Management.Authorization.dll</HintPath>
6363
<Private>True</Private>
6464
</Reference>
65-
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=0.11.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
65+
<Reference Include="Microsoft.Azure.Management.DataLake.Store, Version=0.12.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
6666
<SpecificVersion>False</SpecificVersion>
67-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.0.11.0-preview\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
67+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.Store.0.12.0-preview\lib\net45\Microsoft.Azure.Management.DataLake.Store.dll</HintPath>
6868
<Private>True</Private>
6969
</Reference>
7070
<Reference Include="Microsoft.Azure.Management.DataLake.StoreUploader, Version=0.10.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
7171
<SpecificVersion>False</SpecificVersion>
72-
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.StoreUploader.0.10.1-preview\lib\net45\Microsoft.Azure.Management.DataLake.StoreUploader.dll</HintPath>
72+
<HintPath>..\..\..\packages\Microsoft.Azure.Management.DataLake.StoreUploader.0.10.2-preview\lib\net45\Microsoft.Azure.Management.DataLake.StoreUploader.dll</HintPath>
7373
<Private>True</Private>
7474
</Reference>
7575
<Reference Include="Microsoft.Azure.ResourceManager, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">

0 commit comments

Comments
 (0)