Re-record Batch tests #6405

matthchr · 2018-06-06T16:33:31Z

Also label tests as "CheckIn" category.

Description

See title

Checklist

I have read the Submitting Changes section of CONTRIBUTING.md
The title of the PR is clear and informative
The appropriate change log has been updated
The PR does not introduce breaking changes
If applicable, the changes made in the PR have proper test coverage
For public API changes to cmdlets:
- the changes have gone through a cmdlet design review
- the cmdlet markdown files were generated using the platyPS module

matthchr · 2018-06-06T16:35:10Z

This PR supplants PR #6249

maddieclayton · 2018-06-06T16:40:34Z

src/ResourceManager/AzureBatch/Commands.Batch.Test/ScenarioTests/ScenarioTestHelpers.cs

@@ -507,7 +507,7 @@ public static void WaitForIdleComputeNode(BatchController controller, BatchAccou
                Select = "id,state"
            };

-            DateTime timeout = DateTime.Now.AddMinutes(5);
+            DateTime timeout = DateTime.Now.AddMinutes(10);


Can we change this to 3 hours?

I wanted to see where the tests failed in CI, and then fix up all those places, as I am not sure this is the only place we wait.

I will amend the PR to wait a longer time once I've seen the initial results though, yes.

matthchr · 2018-06-06T16:50:42Z

This fixes issue: #5006

matthchr · 2018-06-06T17:45:25Z

It looks like the test pass in Travis failed because of this error:

[xUnit.net 00:00:20.4340168]     Microsoft.Azure.Commands.Batch.Test.ScenarioTests.CertificateTests.TestCertificateCrudOperations [FAIL]
Failed   Microsoft.Azure.Commands.Batch.Test.ScenarioTests.CertificateTests.TestCertificateCrudOperations
Error Message:
 System.Management.Automation.CmdletInvocationException : Exception calling ".ctor" with "1" argument(s): "error:2006D080:BIO routines:BIO_new_file:no such file"
---- System.Management.Automation.MethodInvocationException : Exception calling ".ctor" with "1" argument(s): "error:2006D080:BIO routines:BIO_new_file:no such file"
-------- Interop+Crypto+OpenSslCryptographicException : error:2006D080:BIO routines:BIO_new_file:no such file
Stack Trace:
   at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.Runspaces.Pipeline.Invoke()
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke()
   at Microsoft.WindowsAzure.Commands.ScenarioTest.EnvironmentSetupHelper.RunPowerShellTest(String[] scripts) in /home/travis/build/Azure/azure-powershell/src/ResourceManager/Common/Commands.ScenarioTests.ResourceManager.Common/EnvironmentSetupHelper.cs:line 638
   at Microsoft.Azure.Commands.Batch.Test.ScenarioTests.BatchController.RunPsTestWorkflow(Func`1 scriptBuilder, Action initialize, Action cleanup, String callingClassType, String mockName) in /home/travis/build/Azure/azure-powershell/src/ResourceManager/AzureBatch/Commands.Batch.Test/ScenarioTests/BatchController.cs:line 112
   at Microsoft.Azure.Commands.Batch.Test.ScenarioTests.BatchController.RunPsTest(String[] scripts) in /home/travis/build/Azure/azure-powershell/src/ResourceManager/AzureBatch/Commands.Batch.Test/ScenarioTests/BatchController.cs:line 55
   at Microsoft.Azure.Commands.Batch.Test.ScenarioTests.CertificateTests.TestCertificateCrudOperations() in /home/travis/build/Azure/azure-powershell/src/ResourceManager/AzureBatch/Commands.Batch.Test/ScenarioTests/CertificateTests.cs:line 34
----- Inner Stack Trace -----
   at System.Management.Automation.DotNetAdapter.AuxiliaryConstructorInvoke(MethodInformation methodInformation, Object[] arguments, Object[] originalArguments)
   at System.Management.Automation.DotNetAdapter.InvokeResolvedConstructor(MethodInformation bestMethod, Object[] newArguments, Object[] arguments)
   at System.Management.Automation.DotNetAdapter.ConstructorInvokeDotNet(Type type, ConstructorInfo[] constructors, Object[] arguments)
   at Microsoft.PowerShell.Commands.NewObjectCommand.CallConstructor(Type type, ConstructorInfo[] constructors, Object[] args)
----- Inner Stack Trace -----
   at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
   at Internal.Cryptography.Pal.CertificatePal.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName)

I think it's complaining because we have a file Resources\BatchTestCert01.cer that it is trying to use but can't find?

I see this issue: https://github.com/dotnet/corefx/issues/11042 suggests the same as above (file not found).

But this same test passes in Jenkins. Do we explicitly exclude .cer files from the travis run somehow?

maddieclayton · 2018-06-06T23:53:47Z

@MiYanni Can you take a look at this?

MiYanni · 2018-06-07T16:57:18Z

@matthchr I just had the same issue for another service's tests. The fix here: 4ab7861

Basically, the path resolution seems to vary depending on if you are running the test via the console runner or the Visual Studio runner. Just do the logic I did in green where you have both potential paths and check to see if one exists; otherwise, use the other path.

maddieclayton

Looks great to me! If it passes all tests I will merge.

maddieclayton · 2018-06-07T19:49:26Z

on demand (to ensure all batch tests are successfully moved): https://azuresdkci.westus2.cloudapp.azure.com/view/PowerShell/job/powershell-demand/920/

maddieclayton · 2018-06-07T21:36:24Z

@matthchr @MiYanni Looks like the test is still failing in travis.

matthchr · 2018-06-07T21:53:50Z

@MiYanni - Can you give specific guidance on how to do your fix in a Powershell context rather than a C# one?

I tried:

    $certPathVs = $localDir + "\Resources\BatchTestCert01.cer"
    $certPathConsole = [System.IO.Path]::GetDirectoryName($PSScriptRoot) + "\Resources\BatchTestCert01.cer"
    $certPath = if (Test-Path $certPathVs -PathType Leaf) { $certPathVs } Else { $certPathConsole }
    $x509cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList $certPath

But that doesn't seem to work.

I assume the issue is the certPathConsole path... what is the directory structure in Travis?

MiYanni · 2018-06-07T23:21:24Z

@matthchr Oh. Your issue isn't that the directory is wrong. Just put:

$x509cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList './Resources/BatchTestCert01.cer'

You can't use backslashes in Linux.

I've had to change every filepath in code to use forward slashes instead. Or use Path.Combine in C# or Join-Path in PowerShell.

matthchr · 2018-06-07T23:23:20Z

Hah! Good catch. Will fix.

- Label tests as "CheckIn" category. - Have a large timeout when in replay mode to deal with possibility of thread starvation in massively parallel run.

matthchr · 2018-06-08T01:00:53Z

There we go, Travis passes now. Feel free to call another on-demand build to make sure things are looking okay.

maddieclayton · 2018-06-08T03:08:02Z

On demand will be fine as I confirmed all batch tests were moved. Merging now! I'll close the flaky test issue now, but let you know if it ever shows back up.

Thanks Matt! We really appreciate your work fixing these tests.

matthchr mentioned this pull request Jun 6, 2018

Update api-version #6249

Closed

8 tasks

maddieclayton suggested changes Jun 6, 2018

View reviewed changes

maddieclayton added the needs-revision label Jun 6, 2018

maddieclayton assigned matthchr Jun 6, 2018

azuresdkci added the needs-review label Jun 6, 2018

matthchr force-pushed the matthchr/fix-tests branch from c8b65d3 to d265472 Compare June 6, 2018 23:06

maddieclayton removed the needs-revision label Jun 6, 2018

maddieclayton assigned MiYanni and unassigned matthchr Jun 6, 2018

matthchr force-pushed the matthchr/fix-tests branch from d265472 to e404dd2 Compare June 7, 2018 19:43

maddieclayton previously approved these changes Jun 7, 2018

View reviewed changes

maddieclayton added Waiting for On Demand and removed needs-review labels Jun 7, 2018

maddieclayton unassigned MiYanni Jun 7, 2018

Re-record Batch tests

6451e93

- Label tests as "CheckIn" category. - Have a large timeout when in replay mode to deal with possibility of thread starvation in massively parallel run.

matthchr dismissed maddieclayton’s stale review via 6451e93 June 8, 2018 00:21

matthchr force-pushed the matthchr/fix-tests branch from e404dd2 to 6451e93 Compare June 8, 2018 00:21

maddieclayton approved these changes Jun 8, 2018

View reviewed changes

maddieclayton merged commit 4c2d8a9 into Azure:preview Jun 8, 2018

matthchr deleted the matthchr/fix-tests branch September 18, 2018 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Re-record Batch tests #6405

Re-record Batch tests #6405

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

maddieclayton Jun 6, 2018

Uh oh!

matthchr Jun 6, 2018

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

maddieclayton commented Jun 6, 2018

Uh oh!

MiYanni commented Jun 7, 2018 •

edited

Loading

Uh oh!

maddieclayton left a comment

Uh oh!

maddieclayton commented Jun 7, 2018

Uh oh!

maddieclayton commented Jun 7, 2018

Uh oh!

matthchr commented Jun 7, 2018

Uh oh!

MiYanni commented Jun 7, 2018 •

edited

Loading

Uh oh!

matthchr commented Jun 7, 2018

Uh oh!

matthchr commented Jun 8, 2018

Uh oh!

maddieclayton commented Jun 8, 2018

Uh oh!

Uh oh!

Re-record Batch tests #6405

Re-record Batch tests #6405

Uh oh!

Conversation

matthchr commented Jun 6, 2018

Description

Checklist

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

maddieclayton Jun 6, 2018

Choose a reason for hiding this comment

Uh oh!

matthchr Jun 6, 2018

Choose a reason for hiding this comment

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

matthchr commented Jun 6, 2018

Uh oh!

maddieclayton commented Jun 6, 2018

Uh oh!

MiYanni commented Jun 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maddieclayton left a comment

Choose a reason for hiding this comment

Uh oh!

maddieclayton commented Jun 7, 2018

Uh oh!

maddieclayton commented Jun 7, 2018

Uh oh!

matthchr commented Jun 7, 2018

Uh oh!

MiYanni commented Jun 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

You can't use backslashes in Linux.

Uh oh!

matthchr commented Jun 7, 2018

Uh oh!

matthchr commented Jun 8, 2018

Uh oh!

maddieclayton commented Jun 8, 2018

Uh oh!

Uh oh!

MiYanni commented Jun 7, 2018 •

edited

Loading

MiYanni commented Jun 7, 2018 •

edited

Loading