Sample Project Best Practices

The ability to generate sample projects from prompts requires example projects that are serialised and passed to the LLM for one or few-shot learning. This page captures the best practices for the example projects to:

• Ensure consistency
• Avoid the limitations of LLMs

What Success Looks Like

1. Customer submits a sample prompt to create a project
2. Customer approves the creation of the project
3. Customer deploys the project
4. A useful, if not wholly complete, deployment is performed
5. Customer understands the value of Octopus

High-Level Goals

An example project must:

• Not rely on credentials being recreated in the target space
  • No sensitive values will be passed to the AI
• Allow a project deployment or runbook run despite the lack of any targets or valid accounts
• Detect default or invalid credentials, and the lack of targets
  • We expect customers to deploy a sample project or run a runbook without first checking any values
• Display a message with links to documentation as next steps
  • We treat the project deployment or runbook run as a wizard that points customers to the next steps

Best Practices

Feeds

• Projects must only reference packages from one or both of the following feeds:
  • Docker images must come from a feed called GHCR Anonymous with the URL https://ghcrfacade-a6awccayfpcpg4cg.eastus-01.azurewebsites.net
    • This feed is a thin facade over the GHCR Docker API for images in the OctopusSolutionsEngineering organisation
    • This allows for unauthenticated tag queries
    • The facade application is found here
  • Packages must come from a Maven feed called Octopus Maven Feed with the URL http://octopus-sales-public-maven-repo.s3-website-ap-southeast-2.amazonaws.com/snapshot
  • This feed uses a public S3 bucket to expose an anonymous Maven repo

Accounts

• Prefer the use of OIDC accounts
• Use GUIDs of 00000000-0000-0000-0000-000000000000 when creating Azure accounts

Sensitive values

• Use CHANGE ME for all sensitive values.
  • This is the value that all sensitive values are defaulted to during export
  • Any scripts checking for default values must assume sensitive values are set to CHANGE ME

Variables

• All variables must have descriptions
  • The LLM learns from these descriptions and uses them when customising the project
• Variables should always use the #{VariableName} syntax
  • This is because the LLM gets confused with Powershell variables starting with a $ and Terraform interpolation
• Project variables should be prefixed with Project.
  • This prevents name collisions with step templates

Deployment Processes

• The first step in a deployment or runbook process must be a step called Validate Setup that detects default or invalid credentials and prints highlight messages indicating next steps
• Deployments must be allowed with no targets
• If the deployment process relies on targets being present, a step must be included called Print Message When no Targets that detects the absence of targets and prints the next steps
  • Not all platforms have targets i.e. AWS Lambdas, Google Cloud Functions, Azure Container Instances etc
  • If we don't expect any targets to be present, there is no need to warn about the lack of targets
• Deployment processes must deploy Octopub packages if possible:
  • Octopub has been designed to support multiple platforms including Lambdas, Azure Functions, Google Functions, Docker etc
  • Octopub embeds SBOMs for security scanning
• The last step in the deployment process must be called Scan for Vulnerabilities with a script that scans the SBOM associated with the deployed application
• There must be one example of each step in a deployment process
  • The purpose of the example projects is to create a base that customers can easily modify and extend via prompts
  • Having a deployment process deploying multiple applications makes it hard for the LLM to know which combination of steps must be used as an example when deploying a single application.
  • For example, a microservice deployment must include the steps to deploy one microservice
  • This does not mean a deployment needs to be done in a single step. If multiple steps are required to deploy a single application, that is fine. This is about avoiding template projects with duplicate steps that do the same thing for multiple applications.
• Complex scripts must be sourced from a public Git repo or a package
  • LLMs often have trouble recreating complex scripts
  • Octopub includes sample scripts at https://github.com/OctopusSolutionsEngineering/Octopub/tree/main/octopus
• Don't use step templates
  • This is a limitation of the Octopus Terraform provider
• Prefer Powershell scripts
  • Python is not an option because AWS and Azure script steps only support Bash and Powershell
  • Powershell works on all hosted workers
  • Powershell can work on Linux
  • Bash will never be well supported on Windows
• Don't fail a deployment because of predictable errors
  • We treat the deployment process as a wizard that points customers to the next step
• All steps must have descriptions
  • The LLM learns from these descriptions and uses the information when customising the project

Projects

• Set the release versions to #{Octopus.Date.Year}.#{Octopus.Date.Month}.#{Octopus.Date.Day}.i

Lifecycles

• The deployment must use a lifecycle called DevSecOps
• The DevSecOps lifecycle has the environments:
  • Development
  • Test
  • Production
  • Security
• The Development and Security environments automatically deploy new releases
• Steps related to application deployment are excluded from the Security environment

Workers

• Default to the Hosted Ubuntu worker
  • It's faster than the Hosted Windows workers

Container Images

• Prefer the use of the Octopus Labs container images which have the name <platform>-workertools:
  • For example, images called aws-workertools, k8s-workertools, azure-workertools
  • We can iterate quickly on these images as needed

Triggers

• The project must trigger a redeployment of the Security environment daily
  • This effectively scans all production deployments every day for new vulnerabilities

Infrastructure creation and destruction

• Runbooks to create and destroy the infrastructure can be included in a project
• If not, the deployment process must link to documentation that describes how to create the required infrastructure if there are no targets or invalid credentials

Config-as-Code

• Example projects can be saved as Config-as-code, but will always be recreated as database projects

Project Groups

• Create example projects in a project group that describes the platform or cloud to which it is being deployed. Examples are:
  • Kubernetes
  • Azure
  • AWS
  • GCP

Certificates

• Avoid certificates as they are hard to recreate

Scripts

Sometimes the LLMs get confused about random things in the scripts. We may have to edit scripts to avoid LLM issues in preference over "clean" scripts.