Merge pull request #107 from huangpf/clu

Clu

Author: huangpf

build.proj

@@ -185,7 +185,7 @@
     DependsOnTargets="Build;Test" >
     <Exec Command="$(LibraryToolsFolder)\CLU\BuildAndInstallClu.bat" />
     <!--do a simple verification-->
-    <Exec Command="$(LibraryRoot)drop\clurun\win7-x64\azure.bat help" />
+    <Exec Command="$(LibraryRoot)drop\clurun\win7-x64\az.bat help" />
   </Target>
 
   <Target Name="BuildMsBuildTask" DependsOnTargets="RestoreNugetPackages">

clu-getstart.md

@@ -2,10 +2,12 @@
 
 ### Prerequsites
 
-Visual Studio 2015 RTM with ASP.NET. For details, check out the [installation doc](http://docs.asp.net/en/latest/getting-started/installing-on-windows.html). 
+* Visual Studio 2015 RTM with ASP.NET. For details, check out the [installation doc](http://docs.asp.net/en/latest/getting-started/installing-on-windows.html). 
 
 Note, after done, run `dnvm list` command to check the 'coreclr' runtime is installed with right version of `1.0.0-rc1-final`. If not, run `dnvm install 1.0.0-rc1-final -r coreclr -a x64 -p`. Remember always use `-p` flag, so the selection can persist. 
 
+* Get the latest dotnet from "https://azureclu.blob.core.windows.net/tools/dotnet-win-x64.latest.zip", unzip, then add its bin folder to the PATH
+
 ### Project Artifacts
 
 CLUPackages require some additional files to direct generation of indexing, and to provide shortcuts when files are installed.  These files can be copied from the Profile project and updated for each package.   
@@ -23,7 +25,7 @@ CLUPackages require some additional files to direct generation of indexing, and
     | ------------- |:-------------:|
     | CommandAssemblies      | File name of cmdlets assembly(ies) |
     | NounPrefix      | ‘AzureRm’  The part of the cmdlet noun to remove in clu commands|
-    | NounFirst       | if true, the verb comes at the end of the command (e.g. azure resource get)|
+    | NounFirst       | if true, the verb comes at the end of the command (e.g. az resource get)|
 
 * \<modulename\>.nuspec.template, which contains nuspec format metadata about the package – the base temaplate is in tools\clu\Microsoft.Azure.Commands.nuspec.template.  Here are the special fields defined in this template:
    * %PackageId% - replace with the module name (Microsoft.Azure.Commands.\<rp-name\>)
@@ -50,35 +52,35 @@ CLUPackages require some additional files to direct generation of indexing, and
    1. Run `<repo-root>\tools\CLU\BuildAndInstallClu.bat` which build and generate all cmdlet packages and deploy to under `<repo root>\drop\clurun` folder, with 3 flavors `win7-x64`, `osx.10.10-x64` and `ubuntu.14.04-x64`. When you have a clean environment or just pull from upstream, you should clean temporary bits such as `git clean -xdf`, and run this command.
    2. Run `<repo-root>\tools\CLU\BuildCmdlet <package name like Microsoft.Azure.Commands.Profile>` <name like: Microsoft.Azure.Commands.Profile>", this will build and refresh an individual cmdlet package.
 
-After #1 above is finished, you can run `drop\clurun\<platform>\azure.bat help` to explore.
+After #1 above is finished, you can run `drop\clurun\<platform>\az.bat help` to explore.
 
 To debug, set environment variable of `DebugCLU` to "1". Then on running any command, you will be prompted to attach a debugger.
 
-There is also `<repo-root>\tools\CLU\SetupEnv.bat` which is a windows batch wrapping around the `BuildAndInstallClu.bat`, plus set the `DebugCLU` for you, and add the `drop\clurun\win7-x64\azure.bat` to the PATH environment variable.
+There is also `<repo-root>\tools\CLU\SetupEnv.bat` which is a windows batch wrapping around the `BuildAndInstallClu.bat`, plus set the `DebugCLU` for you, and add the `drop\clurun\win7-x64\az.bat` to the PATH environment variable.
 
-To test on osx/linux boxes, do #1, open `<repo-root>\drop\clurun`, copy the flavor folder to your target machine, and run the "azure.sh" inside. Make sure set execution permission using `chmod +x azure.sh clurun`
+To test on osx/linux boxes, do #1, open `<repo-root>\drop\clurun`, copy the flavor folder to your target machine, and run the "az.sh" inside. Make sure set execution permission using `chmod +x az.sh clurun`
 
 (All of those are subject to change, contact yugangw or adxsdkdev for any questions)
 
 ### Quick introductions on cmdlets
-  *  Run commands using the ‘azure’ prefix, cmdlet nouns, and cmdlet verbs, for example, `azure environment get` maps to the cmdlet `Get-AzureRmEnvironment`
-  *  Cmdlet parameters use the double dash (--) so for example, getting a subscription with a particular name would be: `azure subscription get –-SubscriptionName “name of subscription"`
+  *  Run commands using the ‘az’ prefix, cmdlet nouns, and cmdlet verbs, for example, `az env get` maps to the cmdlet `Get-AzureRmEnvironment`
+  *  Cmdlet parameters use the double dash (--) so for example, getting a subscription with a particular name would be: `az subscription get –-SubscriptionName “name of subscription"`
   * To log in, 3 options
-    * login interactively using device flow, this is the only option for msa account or any org-id with 2fa enforced, example: `azure account add`
-    * login with user and password, this works on org-id w/o 2fa enforced, example: `azure account add --Username [email protected]  --Password password1`
-    * login as service principal. Example: `azure account add --ServicePrincipal --TenantId <tenant> --ApplicationId <id> --Secret <secret>`
+    * login interactively using device flow, this is the only option for msa account or any org-id with 2fa enforced, example: `az account add`
+    * login with user and password, this works on org-id w/o 2fa enforced, example: `az account add --Username [email protected]  --Password password1`
+    * login as service principal. Example: `az account add --ServicePrincipal --TenantId <tenant> --ApplicationId <id> --Secret <secret>`
   * Piping between cmdlets should work the same way that Powerhell piping works
-    ```azure subscription get --SubscriptionName | azure context set```
+    ```az subscription get --SubscriptionName | az context set```
   * You can capture piped output using redirection to a file - the result will be the json serialization of the output object.
-    ```azure subscription get > subscriptions.json```
-  * You can use file input tu aparameter using '@' notation:
-    ```azure command --param1 @file1.json```
+    ```az subscription get > subscriptions.json```
+  * You can use file input to a parameter using '@' notation:
+    ```az command --param1 @file1.json```
     Reads input from file1.json and attempts to deserialize the .net object that is the Parameter type for ```param1```
-    ```azure command --param1 @@file1.json```
+    ```az command --param1 @@file1.json```
     Does the same thing, but treats the input from ```file1.json``` as if it come from the pipeline, so that multiple objects will result in multiple invocations of ```ProcessRecord()``` for the target cmdlet.
   * There are some known issues with the current approach to sessions, which can cause session variables to not be propagated when running cmdlets in a pipeline, to work around this, set the 'CmdletSessionId' environment variable to a numeric value - all cmdlets running from the shell will use that session id, and sessions will work with pipelining 
 
-    ```set CmdletSessionId=1010 ```
+    ```set AzureProfile=1010 ```
 
 ### Testing Cmdlets
 
@@ -92,12 +94,33 @@ Testing will consist of scenario tests and unit tests. Scenario tests should be
 #### Scenario Tests
 - Scenario tests should be saved under `./examples` directory with one directory per package. Each scenario tests should (eventually) consist of both `.ps1` and `.sh` files and should cover "P0" scenarios.
 
+##### Environment Variables for Authentication
+Please set the environment variables for either Username/Password (no 2FA) or ServicePrincipal authentication:
+
+**Username/Password (without 2-factor auth):**
+
+|  Field (case sensitive) |  Description  |
+| ------------- |:-------------|
+| azureUser | an OrgId user name |
+| password | a service principal name |
+| userSubscription | (optional) Selects a particular subscription by id.  If not provided, the first listed subscription will be selected |
+
+**Service Principal:**
+
+|  Field (case sensitive) |  Description  |
+| ------------- |:-------------|
+| spn | The tenant guid to authenticate against |
+| secret | the password or application secret to sue for authentication |
+| tenant | The tenant guid to authenticate against |
+| spnSubscription | (optional) Selects a particular subscription by id.  If not provided, the first listed subscription will be selected |
+
 ##### XUnit Automation For Bash Scenario Tests
 - The ```Commands.Common.ScenarioTest``` project contains classes that enable executing bash scenario tests in Visual Studio, or cross-platform using dnx.
 
 - To implement an xunit bash scenario test you must
   - Add a ```[Collection("SampleCollection")]``` attribute to your test class
-  - Add a field to your class of type ```ScenarioTestFixture``` and add a constructor that initializes it
+  - Add a field to your class of type ```ScenarioTestFixture``` and add a constructor that initializes it.
+  
     ```C#
     [Collection("SampleCollection")]
     public class SampleTestClass
@@ -116,20 +139,22 @@ Testing will consist of scenario tests and unit tests. Scenario tests should be
         _fixture.GetRunner("resource-management").RunScript("01-ResourceGroups");
     }
     ```
-    - Set the environment variable 'TestCredentials' to a connection string providing the credentials to use during test execution. Possible fields include:
-    
-      |  Field (case sensitive) |  Description  |
-      | ------------- |:-------------|
-      |  Username     | an OrgId user name |
-      | ServicePrincipal | a service principal name |
-      | Password      | the password or application secret to sue for authentication |
-      | TenantId      | (required for Service authentication) The tenant guid to authenticate against |
-      | SubscriptionId | (optional) Selects a particular subscription by id.  If not provided, the first listed subscription will be selected |
-    - The infrastructure automatically generates a resource group name and assigns the value to the bash variable ```"$resourceGroupName"```.  If your scripts require additional variables, you can add these to your environment before running tests, or you can generate values using the ScriptRunner (for the tests using that runner).
+    - Set the [environment variables](#Environment_Variables_for_Authentication) for either Username/Password (no 2FA) or ServicePrincipal authentication
+    - Update PATH to include location of CLU bin drop.
+    ```bash
+   export PATH=/<path-to-drop>/clurun/win7-x64/:$PATH
+    ```
+    - The infrastructure automatically generates the following environment variables:
+	   - `BASEDIR` - directory path where test script is located
+	   - `location` - default "WestUS" location 
+	   - `groupName` - randomly generated resource group name (note: the test guarantees that this resource group is deleted at the end of a test run; any other resource groups generated as part of the test run need to be deleted by the test)
+	   - `storageAccountType` - default "Standard_GRS" storage account type 
+	   - `storageAccountName` - randomly generated storage account name 
+   - If the script require additional variables, you can add these to your environment before running tests, or you can generate values using the ScriptRunner (for the tests using that runner).
     ```C#
-        runner.EnvironmentVariables.Add("myVariableName", runner.GenerateName("myres"));
+runner.EnvironmentVariables.Add("myVariableName", runner.GenerateName("myres"));
     ```
-    - Tests can be executed in vs, or by runnign ```dnx test project.json```.  If you execute dnx test from the project directory, it will work without modification and a log file for each script will be written to the test results directory ```..\TestResults```.  If you execute dnx test from a different directory, you must set the following environment variables to provide the path to the examples directory and where to write log files:
+    - Tests can be executed in Visual Studio, or by running ```dnx test project.json```.  If you execute dnx test from the project directory, it will work without modification and a log file for each script will be written to the test results directory ```..\TestResults```.  If you execute dnx test from a different directory, you must set the following environment variables to provide the path to the examples directory and where to write log files:
 
       |  Environment Variable  |  Description  |
       | ------------- |:-------------|
@@ -138,14 +163,14 @@ Testing will consist of scenario tests and unit tests. Scenario tests should be
 
 ##### Running Bash Tests using Bash shell
 - Bash tests should be runnable from bash shell in windows/linux/mac environments.
-- To manually run the tests; please set the following envt. variables for authentication and run `./examples/lib/testrunner.sh`
+- To manually run the tests; please set [environment variables](#Environment_Variables_for_Authentication) for authentication as well as update PATH and run `./examples/lib/testrunner.sh`
+
    ```bash
-   export azureuser=<[email protected]>
-   export azurepassword=<your_password>
-   export PATH=$PATH:/<path-to-drop>/clurun/win7-x64/
-   . /examples/lib/testrunner.sh
+   export azureUser=<[email protected]>
+   export password=<your_password>
+   export PATH=/<path-to-drop>/clurun/win7-x64/:$PATH
    ```
-- All the parameters to the cmdlets should be passed in as envt. variables
+- All the parameters to the cmdlets should be passed in as environment variables
 - The current test runners will provide a unique resource group name via `$groupName` but may not remove it at the end if the test fails.
 - The location for ARM will be provided via variable `$location`.
 - "jq" package and BASH assert (e.g. `[ "foo" == "bar" ]`) should be used to validate the responses.

examples/compute-management/01-VirtualMachineSizes.sh

@@ -1,34 +1,40 @@
-#!/bin/bash
+#!/bin/env/bash
 set -e
 printf "\n=== Managing Virtual Machine Sizes in Azure Compute ===\n"
 
-printf "\nShowing VM size results in location: %s.\n" "$location"
-azure vmsize get --location "$location"
+printf "\n1. Showing VM size results in location: %s.\n" "$location"
+az vm size ls --location "$location"
 
-printf "\nChecking VM size results in location: %s.\n" "$location"
-vmSizeResult=`azure vmsize get --location "$location"`
+printf "\n2. Checking VM size results in location: %s.\n" "$location"
+vmSizeResult=`az vm size ls --location "$location"`
 
-if [ "$vmSizeResult" = "" ]; then
+if [[ $vmSizeResult == "" ]]; then
     echo "Failure: No VM sizes!" 1>&2
     exit 1
 else
-    printf "\nSuccess: Non-empty Results.\n"
+    echo "Success: Non-empty Results."
 fi
 
-queryString=Standard_A0
-result=`echo "$vmSizeResult" | grep -q "$queryString"`
-if [ "$vmSizeResult" = "" ] ; then
-    printf "\nFailure: VM Size Not Found: '%s'.\n" "$queryString"
+filterResult=`az vm size ls --location "$location" | cat | jq 'select(.name | contains("Standard_A0"))' --raw-output`
+if [[ "$filterResult" == "" ]]; then
+    echo "Failure: Standard_A0 vm size not found." 1>&2
     exit 1
 else
-    printf "\nSuccess: VM Size Found in Results: '%s'.\n" "$queryString"
+    echo "Success: Standard_A0 vm size found."
 fi
 
-queryString=Standard_G1
-result=`echo "$vmSizeResult" | grep -q "$queryString"`
-if [ "$vmSizeResult" = "" ] ; then
-    printf "\nFailure: VM Size Not Found: '%s'.\n" "$queryString"
+filterResult=`az vm size ls --location "$location" | cat | jq 'select(.name | contains("Standard_G1"))' --raw-output`
+if [[ "$filterResult" == "" ]]; then
+    echo "Failure: Standard_G1 vm size not found." 1>&2
     exit 1
 else
-    printf "\nSuccess: VM Size Found in Results: '%s'.\n" "$queryString"
+    echo "Success: Standard_G1 vm size found."
 fi
+
+filterResult=`az vm size ls --location "$location" | cat | jq 'select(.name | contains("NonStandard_A1"))' --raw-output`
+if [[ "$filterResult" == "" ]]; then
+    echo "Success: NonStandard_A1 vm size not found."
+else
+    echo "Failure: NonStandard_A1 vm size found." 1>&2
+    exit 1
+fi 

examples/compute-management/02-VirtualMachineCreation.ps1

@@ -0,0 +1,6 @@
+Param(
+  [string]$resourceGroupName,
+  [string]$resourceGroupLocation
+)
+
+Write-Host "Skip"

examples/compute-management/02-VirtualMachineCreation.sh

@@ -0,0 +1,36 @@
+#!/bin/env/bash
+set -e
+printf "\n=== Managing Virtual Machine Creation in Azure Compute ===\n"
+
+printf "\n1. Creating a new resource group: %s and location: %s.\n" "$groupName" "$location"
+az resourcemanager group create -n "$groupName" --location "$location"
+
+printf "\n2. Creating a new storage account '%s' in type '%s'.\n" "$storageAccountName" "$storageAccountType"
+az storage account create--resourcegroupname "$groupName" --name "$storageAccountName" --location "$location" --type "$storageAccountType"
+
+printf "\n3. Create virtual network.\n"
+result=`az vnet create--resourcegroupname "$groupName" --name test --location "$location" --addressprefix "[\"10.0.0.0/16\"]" --subnet "[{\"Name\":\"test\",\"AddressPrefix\":\"10.0.0.0/24\"}]" --force`
+
+contextResult=`az context ls`
+
+subId=`echo $contextResult | jq '.Subscription.SubscriptionId' --raw-output`
+
+subnetId="/subscriptions/$subId/resourceGroups/$groupName/providers/Microsoft.Network/virtualNetworks/test/subnets/test"
+
+printf "\n4. Create network interface with:\r\nsubId='%s' \r\n& \r\nsubnetId='$subnetId'.\n" "$subId"
+export MSYS_NO_PATHCONV=1
+az networkinterface create--name test --resourcegroupname "$groupName" --location "$location" --subnetid "$subnetId"
+export MSYS_NO_PATHCONV=
+
+nicId="/subscriptions/$subId/resourceGroups/$groupName/providers/Microsoft.Network/networkInterfaces/test"
+
+vhdUri="https://$storageAccountName.blob.core.windows.net/$storageAccountName/$storageAccountName.vhd"
+
+vmStr="{\"Name\":\"test\",\"HardwareProfile\":{\"VmSize\":\"Standard_A1\"},\"NetworkProfile\":{\"NetworkInterfaces\":[{\"Id\":\"$nicId\"}]},\"OSProfile\":{\"ComputerName\":\"test\",\"AdminPassword\":\"BaR@1234\",\"AdminUsername\":\"Foo12\"},\"StorageProfile\":{\"ImageReference\":{\"Offer\":\"WindowsServer\",\"Publisher\":\"MicrosoftWindowsServer\",\"Sku\":\"2008-R2-SP1\",\"Version\":\"latest\"},\"OSDisk\":{\"Caching\":\"ReadWrite\",\"CreateOption\":\"FromImage\",\"Name\":\"osDisk\",\"Vhd\":{\"Uri\":\"$vhdUri\"}}}}"
+
+printf "\n5. Create virtual machine with\r\nnicId='%s'\r\nvhdUri='%s'\r\nvmStr='%s'\n" "$nicId" "$vhdUri" "$vmStr"
+
+az vm create--resourcegroupname "$groupName" --location "$location" --vmprofile "$vmStr"
+
+printf "\n6. Removing resource group: %s.\n" "$groupName"
+az resourcemanager group rm -n "$groupName" -f

examples/lib/helper.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/bin/env/bash
 
 randomName() {
     echo "$1$RANDOM"

examples/lib/loginService.sh

@@ -1,2 +1,2 @@
-#!/bin/bash
-azure account add --spn --appid "$spn" --secret "$secret" -t "$tenant" -s "$subscription"
+#!/bin/env/bash
+az login --spn --appid "$spn" --secret "$secret" -t "$tenant" -s "$spnSubscription"

examples/lib/loginUser.sh

@@ -1,2 +1,2 @@
-#!/bin/bash
-azure account add -u "$azureUser" -p "$password" -s "$subscription"
+#!/bin/env/bash
+az login -u "$azureUser" -p "$password" -s "$userSubscription"

examples/lib/testrunner.sh

@@ -1,21 +1,21 @@
-#!/bin/bash
-export BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
-. $BASEDIR/helper.sh
+#!/bin/env/bash
+export TESTDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+. $TESTDIR/helper.sh
 export groupName=`randomName testrg`
 export location="westus"
-export CmdletSessionID=1010
 export MSYS_NO_PATHCONV=1
 
 echo "Logging in as user"
-. $BASEDIR/loginUser.sh
+. $TESTDIR/loginUser.sh
 
-for d in $( ls $BASEDIR/.. --ignore=lib ); do
-    for f in $( ls $BASEDIR/../$d/*.sh ); do
+for d in $( ls $TESTDIR/.. --ignore=lib ); do
+    for f in $( ls $TESTDIR/../$d/*.sh ); do
         echo "running: $f"
+        BASEDIR=$(cd "$(dirname "$f")" && pwd)
         . $f
         set +e
         printf "\nCleanup: removing resource group: %s\n" $groupName
-        azure group remove --name "$groupName" --force
+        az resourcemanager group rm --name "$groupName" --force
         set -e
         echo "success: $f"
     done