Tests for chef extension arm commands

[NimishaS]

Test suite for #2587

w.r.t. #2638

[azurecla]

Hi @NimishaS, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.

TTYL, AZPRBOT;

[NimishaS]

@markcowl , @hovsepm, I am getting this error while running the tests in Record mode:

Test Name:  Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ChefExtensionTests.TestChefExtensionBasic
Test FullName:  Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ChefExtensionTests.TestChefExtensionBasic
Test Source:    c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\ScenarioTests\ChefExtensionTests.cs : line 11
Test Outcome:   Failed
Test Duration:  0:00:00.001

Result Message: System.IO.FileLoadException : Could not load file or assembly 'Microsoft.Rest.ClientRuntime.Azure.Authentication, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. Strong name signature could not be verified.  The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045)
Result StackTrace:  
at Microsoft.Azure.Test.TokenCloudCredentialsHelper.GetUserToken(String authEndpoint, String tenant, String clientId, String managementResource, String graphResource, String userName, String password)
   at Microsoft.Azure.Test.Authentication.OrgIdAuthenticationProvider.TryDoAuthentication(TestEnvironment env, IDictionary`2 settings, ExecutionMode mode, AuthorizationContext& context) in D:\workspace\test-framework-publish\Test.Framework\Authentication\OrgIdAuthenticationProvider.cs:line 31
   at Microsoft.Azure.Test.Authentication.AuthenticationProvider.Authenticate(TestEnvironment environment, IDictionary`2 settings, ExecutionMode mode) in D:\workspace\test-framework-publish\Test.Framework\Authentication\AuthenticationProvider.cs:line 28
   at Microsoft.Azure.Test.Authentication.AuthenticationProvider.Authenticate(TestEnvironment environment, IDictionary`2 settings, ExecutionMode mode) in D:\workspace\test-framework-publish\Test.Framework\Authentication\AuthenticationProvider.cs:line 30
   at Microsoft.Azure.Test.Authentication.AuthenticationProvider.Authenticate(TestEnvironment environment, IDictionary`2 settings, ExecutionMode mode) in D:\workspace\test-framework-publish\Test.Framework\Authentication\AuthenticationProvider.cs:line 30
   at Microsoft.Azure.Test.TestEnvironmentFactory.LoadTestEnvironment(String envVariableName, ExecutionMode mode) in D:\workspace\test-framework-publish\Test.Framework\TestEnvironmentFactory.cs:line 117
   at Microsoft.Azure.Test.CSMTestEnvironmentFactory.GetTestEnvironmentFromContext() in D:\workspace\test-framework-publish\Test.Framework\CSMTestEnvironmentFactory.cs:line 37
   at Microsoft.Azure.Test.TestEnvironmentFactory.GetTestEnvironment() in D:\workspace\test-framework-publish\Test.Framework\TestEnvironmentFactory.cs:line 73
   at Microsoft.Azure.Test.TestBase.GetServiceClient[T](TestEnvironmentFactory factory) in D:\workspace\test-framework-publish\Test.Framework\TestBase.cs:line 47
   at Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ComputeTestController.GetResourceManagementClient() in c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\Common\ComputeTestController.cs:line 237
   at Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ComputeTestController.SetupManagementClients(MockContext context) in c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\Common\ComputeTestController.cs:line 180
   at Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ComputeTestController.RunPsTestWorkflow(Func`1 scriptBuilder, Action`1 initialize, Action cleanup, String callingClassType, String mockName) in c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\Common\ComputeTestController.cs:line 136
   at Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ComputeTestController.RunPsTest(String[] scripts) in c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\Common\ComputeTestController.cs:line 100
   at Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ChefExtensionTests.TestChefExtensionBasic() in c:\Users\azure\nim\azure-powershell\src\ResourceManager\Compute\Commands.Compute.Test\ScenarioTests\ChefExtensionTests.cs:line 12

[NimishaS]

I have specified the following environment variables:

AZURE_LIVEID : UserId=<[email protected]>;Password=<Password>  #Provided azure account credentials

AZURE_SERVICE_PRINCIPAL : UserId=<UserGuid>;Password=<password>;AADTenant=<TenantGuid>;SubscriptionId=<SubscriptionId>  #provided azure account credentials in UserId and Password

AZURE_TEST_MODE : Record

TEST_CSM_ORGID_AUTHENTICATION : SubscriptionId={subscription-id};BaseUri=https://api-next.resources.windows-int.net/;UserId={user-id};Password={password}   #provided azure account credentials in UserId and Password

TEST_ORGID_AUTHENTICATION : SubscriptionId={subscriptionid};BaseUri=https://management.core.windows.net/;AADAuthEndpoint=https://login.windows.net/;GraphUri=https://graph.windows.net/

Please let us know if we missed anything.

[markcowl]

@NimishaS This is most likely due to dependency mismatch ensure that the dependencies of your libraries are up to date and match the dependencies of the test framework. If you look at the dev branch, the test framework and other versions there are correct.

[NimishaS]

@markcowl , tried the following approaches:
a) Updating all the packages by running Update-Package from Package Manager Console. Build fails after this.
b) Manually installing Microsoft.Rest.ClientRuntime.Azure.Authentication package:

Install-Package Microsoft.Rest.ClientRuntime.Azure.Authentication -Version 2.2.7-preview

c) Also tried nuget install packages.config for both Commands.Compute and Commands.Compute.Test. It says All listed packages in config are already installed.
Still facing the same issue.

Any pointers that can help?

[shahabhijeet]

@NimishaS Please pull from dev branch and update the following library Microsoft.Rest.ClientRuntime.Azure.Authentication (ver 2.2.8) and the test framework and then try to run your test.

[NimishaS]

@markcowl , @shahabhijeet the library related issues got resolved. I am facing an issue at this line now.

The term 'Test-SetChefExtensionBasic' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I am missing some link. How is Test-SetChefExtensionBasic made available here? Do I have to add some path to the PS file where this method is defined?
I am referring DiagnosticExtensionTests, but I couldn't find any reference for Test-DiagnosticsExtensionBasic method or DiagnosticExtensionTests.ps1 file added anywhere.

[mwrock]

looks like you are missing <CopyToOutputDirectory>Always</CopyToOutputDirectory> in Commands.Compute.Test.csproj so when building the project the ps1 file is missing from the output where the tests need to load it from.

[NimishaS]

Thanks for help @mwrock. My specs are failing at Create-VirtualMachine method in ChefExtensionTests.ps1 file now.
Tests run fine with this much code in ChefExtensionTests.ps1 file:

$rgname = Get-ComputeTestResourceName
$loc = Get-ComputeVMLocation

And they fail on including Create-VirtualMachine method

[NimishaS]

@markcowl, we are currently facing an issue where the AzureRM* modules are not getting loaded while running the test in record mode. That's why Create-VirtualMachine method is failing. The method works fine when run manually.

[NimishaS]

@markcowl , I have added the recorded tests response in the SessionRecords. Verified that it's executing successfully in playback mode too.
Still travis is failing. We don't have permission to check the details. Can you please share the error?

[NimishaS]

@markcowl , @hovsepm , please share the details of Travis failure. We don't have the required permissions.
Recorded test response is pushed now.

[NimishaS]

@markcowl , @hovsepm, we are tracking this as one of our tasks. Please let us know what's missing in the PR.

[shahabhijeet]

@NimishaS one of your tests is failing, could you take a look at it. Also could you confirm if you do not see this failure locally when you run the test?

5.937301s ✘  Microsoft.Azure.Commands.Compute.Test.ScenarioTests.ChefExtensionTests.TestSetChefExtensionBasic
System.Management.Automation.ActionPreferenceStopException : The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Could not find a part of the path 'D:\workspace\powershell\src\ResourceManager\Network\Commands.Network.Test\bin\Debug\Templates\client.rb'.
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input, Hashtable errorResults, Boolean enumerate)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)

[NimishaS]

@shahabhijeet , I don't get this error while running the test locally in Playback mode. The tests are executing successfully.
Also the error that you shared is coming in D:\workspace\powershell\src\ResourceManager\Network\Commands.Network.Test\bin\Debug\Templates\client.rb, which should ideally not be affected by this code.

[shahabhijeet]

@NimishaS can you verify if you now have access to the CI server and can see the logs?

[NimishaS]

@shahabhijeet I still can't access the CI server to see the logs

[shahabhijeet]

@NimishaS Can you follow the below documentation
https://github.com/Azure/azure-powershell/blob/dev/documentation/Using-Azure-TestFramework.md
https://github.com/Azure/azure-powershell/blob/dev/documentation/Repo-Tasks-Module.md
https://github.com/Azure/azure-powershell/wiki/Azure-Powershell-Developer-Guide

And let me know if you can then run your tests locally and record any tests.

[NimishaS]

@shahabhijeet , tests are executing successfully in playback mode. I just had a careful look at the error raised again (#2713 (comment)):

Could not find a part of the path 'D:\workspace\powershell\src\ResourceManager\Network\Commands.Network.Test\bin\Debug\Templates\client.rb'.
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input, Hashtable errorResults, Boolean enumerate)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)

I have added <CopyToOutputDirectory>Always</CopyToOutputDirectory> for Templates/client.rb file in Commands.Compute.Test.csproj, but still the error raised is:

Could not find a part of the path 'D:\workspace\powershell\src\ResourceManager\Network\Commands.Network.Test\bin\Debug\Templates\client.rb'

Can you please confirm if this has something to do with the contents of the file?

[NimishaS]

Also not sure why Travis is looking for client.rb file at D:\workspace\powershell\src\ResourceManager\Network\Commands.Network.Test\bin\Debug\Templates\client.rb. On my local machine, it's available at D:\workspace\powershell\src\ResourceManager\Compute\Commands.Compute.Test\bin\Debug\Templates.
The changes made are in Compute module.

[cormacpayne]

@shahabhijeet can you take a look?

[markcowl]

@NimishaS This is because tests are running in parallel on the CI, so you cnanot count on the current directory. You can use AppDomain.Current to determine the test directory when the test suit is started, as here: https://github.com/Azure/azure-powershell/blob/dev/src/ResourceManager/KeyVault/Commands.KeyVault.Test/ScenarioTests/KeyVaultManagementTests.cs#L36 and pass this in to you powershell script for any ancillary files it requires

[NimishaS]

@markcowl thanks for help. CI is passing now.