Skip to content

Added -JsonAttributes and -ChefServiceInterval option #3083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

Vasu1105
Copy link

@Vasu1105 Vasu1105 commented Oct 14, 2016

Description

-- Added -JsonAttribute option in Set-AzureVMChefExtension and Set-AzureRMVMChefExtension command. It sets the custom json attributes in first-boot.json.
-- Added -ChefServiceInterval option in Set-AzureVMChefExtension and Set-AzureRMVMChefExtension command.Specifies the frequency (in minutes) at which the chef-service runs. If in case you don't want the chef-service to be installed on the Azure VM then set value as 0 in this field.

This checklist is used to make sure that common guidelines for a pull request are followed. You can find a more complete discussion of PowerShell cmdlet best practices here.

General Guidelines

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.
  • The pull request does not introduce breaking changes (unless a major version change occurs in the assembly and module).

Testing Guidelines

  • Pull request includes test coverage for the included changes.
  • PowerShell scripts used in tests should do any necessary setup as part of the test or suite setup, and should not use hard-coded values for locations or existing resources.

Cmdlet Signature Guidelines

  • New cmdlets that make changes or have side effects should implement ShouldProcess and have SupportShouldProcess=true specified in the cmdlet attribute. You can find more information on ShouldProcess here.
  • Cmdlet specifies OutputType attribute if any output is produced - if the cmdlet produces no output, it should implement a PassThrough parameter.

Cmdlet Parameter Guidelines

  • Parameter types should not expose types from the management library - complex parameter types should be defined in the module.
  • Complex parameter types are discouraged - a parameter type should be simple types as often as possible. If complex types are used, they should be shallow and easily creatable from a constructor or another cmdlet.
  • Cmdlet parameter sets should be mutually exclusive - each parameter set must have at least one mandatory parameter not in other parameter sets.
    …oot.json

Signed-off-by: Vasu1105 [email protected]

@azurecla
Copy link

Hi @Vasu1105, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
We will now validate the agreement and then real humans will evaluate your PR.

TTYL, AZPRBOT;

@shahabhijeet
Copy link
Contributor

I don't see any tests for the new parameter? Can you add test coverage.

@Vasu1105
Copy link
Author

@shahabhijeet yes working on it

ValueFromPipelineByPropertyName = true,
HelpMessage = "A JSON string to be added to the first run of chef-client.")]
[ValidateNotNullOrEmpty]
public string JsonAttributes { get; set; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use singular for parameter name.

Copy link
Author

@Vasu1105 Vasu1105 Oct 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hyonholee In chef this parameter is referred as --json-attributes and continuing same naming convention as it is the option of chef. Also json attributes can contain multiple attributes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vasu1105 This is a Best Practice guideline for parameter names, please update the parameter name to be in parallel with best practices.

@@ -263,38 +271,86 @@ private Hashtable PublicConfiguration
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change the order of conditions to make the code simpler. Put if(IsRunListEmpty) condition inside of other condition, and you can avoid repetition.

@@ -213,34 +220,77 @@ private void SetPublicConfig()
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above. Change the order of condition and make it simple.

hashTable.Add(BootstrapVersionTemplate, BootstrapVersion);
hashTable.Add(ClientRbTemplate, ClientConfig);
this.publicConfiguration = hashTable;
if (IsJsonAttributesEmpty)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we replace the multiple nested Ifs with a simpler logic as below. The same logic can be applied to while constructing ClientConfig string

private Hashtable PublicConfiguration
{
get
{
if (_publicConfiguration == null)
{
_publicConfiguration = new Hashtable();
_publicConfiguration.Add(ClientRbTemplate, ClientConfig);

                if (!string.IsNullOrEmpty(this.RunList))
                    _publicConfiguration.Add(RunListTemplate, this.RunList);

                if (!string.IsNullOrEmpty(BootstrapVersion))
                    _publicConfiguration.Add(BootstrapVersionTemplate, BootstrapVersion);

                if (!string.IsNullOrEmpty(this.BootstrapOptions))
                    _publicConfiguration.Add(BootStrapOptionsTemplate, this.BootstrapOptions);

                if (!string.IsNullOrEmpty(JsonAttributes))
                    _publicConfiguration.Add(JsonAttributesTemplate, JsonAttributes);

}
return _publicConfiguration
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shahabhijeet Yes I am already doing this change.

ValueFromPipelineByPropertyName = true,
HelpMessage = "A JSON string to be added to the first run of chef-client.")]
[ValidateNotNullOrEmpty]
public string JsonAttributes { get; set; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simply the logic of nested If's with a simpler one to initialize the Hashtable (as stated above in ResourceManager/Compute/../../../SetAzureVMCheckExtension.cs)

Copy link
Author

@Vasu1105 Vasu1105 Oct 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In service management this.PublicConfiguration is string and I am not able to assign Hash to string. I am new to c#. I tried following but I am getting error
` var hashTable = new Hashtable();
hashTable.Add(BootstrapVersionTemplate, this.BootstrapVersion);
hashTable.Add(ClientRbTemplate, ClientConfig);

if (!IsRunListEmpty)
{
hashTable.Add(RunListTemplate, this.RunList);
}

if (!IsBootstrapOptionsEmpty)
{
hashTable.Add(BootStrapOptionsTemplate, this.BootstrapOptions);
}
this.PublicConfiguration = hashTable.ToString(); `

@@ -101,6 +102,12 @@ public string TypeHandlerVersion
public string BootstrapOptions { get; set; }

[Parameter(
ValueFromPipelineByPropertyName = true,
HelpMessage = "A JSON string to be added to the first run of chef-client.")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you be more precise what is the purpose of this Json string? Describe in brief what are you injecting during bootstrapping of the chef-client

@Vasu1105 Vasu1105 changed the title Added -JsonAttributes option to set custom json attributes in first-b… Added -JsonAttributes and -ChefServiceInterval option Oct 20, 2016
@shahabhijeet
Copy link
Contributor

@hyonholee please approve in order to merge.

@hyonholee
Copy link
Contributor

LGTM. Thanks.

ValueFromPipelineByPropertyName = true,
HelpMessage = "A JSON string to be added to the first run of chef-client.")]
[ValidateNotNullOrEmpty]
public string JsonAttributes { get; set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to make this parameter singular per PowerShell Guidelines

@markcowl
Copy link
Member

@Vasu1105 This is marked as [Do Not Merege] in the description, Please let us know if you want to merge these changes.

@markcowl markcowl assigned markcowl and unassigned shahabhijeet Oct 26, 2016
@Vasu1105
Copy link
Author

@markcowl I have fixed the parameter names. This can be merged.

Copy link
Member

@markcowl markcowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vasu1105 I don't see any automated tests for the new parameters. Since this is the last day of code complete, you cna manually verify this and add tests in the next sprint, if you don't have time to automate tests today. Please file an issue in https://github.com/azure/azure-powershell/issues and reference it here and we cna merge

@@ -101,6 +103,18 @@ public string TypeHandlerVersion
public string BootstrapOptions { get; set; }

[Parameter(
ValueFromPipelineByPropertyName = true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any tests for this change

Copy link
Author

@Vasu1105 Vasu1105 Oct 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markcowl I have a dependency on #2173 for adding automated test in Resource management i.e. for Set-AzureRMVMChefExtension so I have created separate issue #3147 for this.

@@ -66,6 +68,18 @@ public class SetAzureVMChefExtensionCommand : VirtualMachineChefExtensionCmdletB
public string RunList { get; set; }

[Parameter(
ValueFromPipelineByPropertyName = true,
HelpMessage = "A JSON string to be added to the first run of chef-client. e.g. -JsonAttribute '{\"foo\" : \"bar\"}'")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests for the change?

Copy link
Author

@Vasu1105 Vasu1105 Oct 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markcowl I have created the issue #3146 since there is no Automated test written for the Set;AzureVMChefExtension module and I have to write it from scratch. Also I have a dependency on #2713 for adding automated test in Resource management i.e. for Set-AzureRMVMChefExtension so I have created separate issue #3147 for this.

@markcowl
Copy link
Member

on demand run here:http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1268/

LGTM once the on-demand run passes

@markcowl markcowl removed their assignment Oct 27, 2016
@markcowl
Copy link
Member

@Vasu1105 PR is failing in the on demand run (see above). Will need to track down this issue before it can be merged. You may just need to rebase your change on the latest commits in the dev branch

@Vasu1105 Vasu1105 force-pushed the vj/adding_json_attributes_option_for_chef_extension branch from 6d8c2ca to cd8c6de Compare October 28, 2016 05:16
@Vasu1105
Copy link
Author

@markcowl I don't have access to this link http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1268/. I have rebased my branch now.

@markcowl
Copy link
Member

markcowl commented Nov 3, 2016

new on demand run here: http://azuresdkci.cloudapp.net/view/1-AzurePowerShell/job/powershell-demand/1290/

Now that this is scheduled for the next release, please add the test coverage before wew merge

Copy link
Member

@markcowl markcowl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Coverage Needed. Now that this didn't make the release, you have until the end of the month to get the test coverage in, please add the test coverage, or let us know why this isn't possible.

@Vasu1105
Copy link
Author

Vasu1105 commented Nov 7, 2016

ok @markcowl I am working on adding automated test for ASM Set-AzureVMChefExtension. As I mentioned I have a dependency on #2713 for adding automated test for newly added option in Resource manager(ARM) Set-AzureVmChefExtension so I won't be able to do that unless that PR gets merged.

@Vasu1105
Copy link
Author

Vasu1105 commented Nov 8, 2016

@shahabhijeet & @markcowl I have added the test and its passing on my local machine. I don't have access due to which unable to see whats failing here.

@markcowl
Copy link
Member

@Vasu1105 Here is the filure, it looks like you didn't include your json recording in the test csproj, and have it marked as 'CopyAlways' or 'CopyIfNewer'. See: https://github.com/Azure/azure-powershell/blob/dev/src/ResourceManager/Compute/Commands.Compute.Test/Commands.Compute.Test.csproj#L283 for an example

StackTrace:

03:11:23 D:\workspace\test-framework-publish\Test.Framework\TestUtilities.cs(217): error : Microsoft.WindowsAzure.Commands.ScenarioTest.ChefExtensionTests.TestSetAzureVMChefExtension [FAIL] [D:\workspace\powershell\build.proj]
03:11:23 System.ArgumentException : Unable to find recorded mock file 'D:\workspace\powershell\src\ServiceManagement\Common\Commands.ScenarioTest\bin\Debug\SessionRecords\Microsoft.WindowsAzure.Commands.ScenarioTest.ChefExtensionTests\RunPowerShellTest.json'.
03:11:23 Parameter name: callerIdentity
03:11:23 Stack Trace:
03:11:23 at Microsoft.Azure.Test.HttpRecorder.HttpMockServer.Initialize(String callerIdentity, String testIdentity, HttpRecorderMode mode)
03:11:23 D:\workspace\test-framework-publish\Test.Framework\TestUtilities.cs(217,0): at Microsoft.Azure.Test.TestUtilities.StartTest(String className, String methodName)
03:11:23 D:\workspace\test-framework-publish\Test.Framework\UndoHandler\UndoContext.cs(123,0): at Microsoft.Azure.Test.UndoContext.Start(String className, String methodName)
03:11:23 src\ServiceManagement\Common\Commands.ScenarioTest\ChefExtension\ChefExtensionTests.cs(55,0): at Microsoft.WindowsAzure.Commands.ScenarioTest.ChefExtensionTests.RunPowerShellTest(String[] scripts)
03:11:23 src\ServiceManagement\Common\Commands.ScenarioTest\ChefExtension\ChefExtensionTests.cs(33,0): at Microsoft.WindowsAzure.Commands.ScenarioTest.ChefExtensionTests.TestSetAzureVMChefExtension()

@Vasu1105
Copy link
Author

Vasu1105 commented Nov 14, 2016

@markcowl in previous failure TestSetAzureVMChefExtension.json was not getting generated and it was looking for RunPowerShellTest.json. I fixed this here f88e5e1#diff-4ee6fa107c88a2c62688a18fe08ff557R55

Now 'Unable to find recorded mock file ' should not come. And as you mentioned about adding test recording json in csproj I have already did that here 411f0a3#diff-fc0f47757ee5e76682344d8e77d84380R280

For me test is running fine in record and playback mode on local.

Could you please let me know whats the error its now throwing.

Its becomes very difficult to check error since we don't have access to logs.

@cormacpayne
Copy link
Member

@shahabhijeet can you take a look?

@markcowl
Copy link
Member

@Vasu1105 The issue is in a reference to a .pem file in your tests. Sicne tests run in parallel on the ci machine, you cannot count on the current directory, and need to refer to the base directory correctly

See: https://github.com/Azure/azure-powershell/blob/dev/src/ResourceManager/KeyVault/Commands.KeyVault.Test/ScenarioTests/KeyVaultManagementTests.cs#L36 for an example


$vm = Get-AzureVM -ServiceName $svcName -Name $vmName

Set-AzureVMChefExtension -VM $vm -ValidationPem ".\Resources\ChefExtension\tstorgnztn-validator.pem" -ClientRb ".\Resources\ChefExtension\client.rb" -JsonAttribute '{"container_service": {"chef-init-test": {"command": "C:\\opscode\\chef\\bin"}}}' -ChefServiceInterval 35 -Windows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue with your tests is that, when runnign tests in parallel, the current directory can change. The best option to fix this is to grab the value of AppDomain.Current.BaseDirectory, and pass this to your script, so you are not dependent on the current directory

@Vasu1105 Vasu1105 force-pushed the vj/adding_json_attributes_option_for_chef_extension branch from cadd2cb to b369cc7 Compare November 22, 2016 09:25
@Vasu1105
Copy link
Author

@markcowl I have updated my script for adding current directory path as suggested by you here b369cc7#diff-1ca6f8ad9bc35ff72b8bbfcbcd491baaR33

After doing above change I have run the specs in record and playback mode both on my local machine and both ran successfully also I pushed my changes and also re-base the branch but still the test are failing.

I referred following existing automated test for this.

https://github.com/Azure/azure-powershell/blob/dev/src/ServiceManagement/Common/Commands.ScenarioTest/Resources/ServiceManagement/ServiceManagementTests.ps1#L726
&
https://github.com/Azure/azure-powershell/blob/dev/src/ServiceManagement/Common/Commands.ScenarioTest/Resources/DiagnosticsExtension/DiagnosticsExtensionTests.ps1#L25

@Vasu1105
Copy link
Author

Vasu1105 commented Nov 22, 2016

@markcowl It will be helpful if we (@NimishaS & me) get access to CI logs so that we can find out what's the error. Currently we need to wait till the feedback comes.

@markcowl
Copy link
Member

@Vasu1105 You can get access to the logs by following the instructions here: http://aka.ms/azuregithub for joinging the azure organization

@markcowl
Copy link
Member

This is now failing due to credential scanner issues: Found match - D:\workspace\powershell\src\ServiceManagement\Common\Commands.ScenarioTest\Resources\ChefExtension\tstorgnztn-validator.pem
02:02:18 Scan successfully completed - Duration: 00:01:24.6061452
02:02:19 Credential scanner match found exception!
02:02:19 At C:\ci-signing\adxsdk\RunCredScan.ps1:71 char:5
02:02:19 + throw "Credential scanner match found exception!"
02:02:19 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You will need to make sure this isn't a real credential used anywhere, and then add an inline exception like this: https://github.com/Azure/azure-powershell/blob/2eaf6175bc2533b83689120dc7f595ff0bdf8001/src/ResourceManager/LogicApp/Commands.LogicApp.Test/SessionRecords/Microsoft.Azure.Commands.LogicApp.Test.ScenarioTests.IntegrationAccountCertificateTests/TestUpdateIntegrationAccountCertificate.json or a global exception for the file here: https://github.com/Azure/azure-powershell/blob/dev/tools/GlobalFilters.xml#L163

@Vasu1105
Copy link
Author

@markcowl Thanks for the help. Now test are passing here.

The link http://aka.ms/azuregithub which you shared for following instruction to get access to logs is not working for me. I am getting azure login page when I am clicking on link, I have used one existing azure account credentials for login but its giving bad request error.

@shahabhijeet
Copy link
Contributor

@shahabhijeet
Copy link
Contributor

@Vasu1105 waiting for @hyonholee to approve final set of changes.

@shahabhijeet
Copy link
Contributor

@Vasu1105 can you add changes in changelog.md following the guidelines
https://github.com/Azure/azure-powershell/blob/dev/CONTRIBUTING.md#updating-the-change-log

Vasu1105 added 9 commits November 30, 2016 09:34
…rce Management, Updated -JsonAttribute description with example, Added new -ChefServiceInterval option

Signed-off-by: Vasu1105 <[email protected]>
…anagement command to reduce multiple conditions

Signed-off-by: Vasu1105 <[email protected]>
Signed-off-by: Vasu1105 <[email protected]>
Signed-off-by: Vasu1105 <[email protected]>
@Vasu1105 Vasu1105 force-pushed the vj/adding_json_attributes_option_for_chef_extension branch from da5b108 to 456a40d Compare November 30, 2016 09:34
@Vasu1105
Copy link
Author

@shahabhijeet I have updated the changelog,md

@hyonholee
Copy link
Contributor

LGTM. Please add test and recorded file.

@Vasu1105
Copy link
Author

Vasu1105 commented Dec 1, 2016

@shahabhijeet & @hyonholee I have added the test and recording files already

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants