Revert 35 revert 33 build docker for e2e

[harrryr]

Issue #, if available:
Had to revert this PR due to syntax error in the publish metric step. Reposting the change with error fixed

Test run: https://github.com/aws-observability/aws-application-signals-test-framework/actions/runs/8621464546
Also checked that the metric is present in the console
Test run after switching to ECR: https://github.com/aws-observability/aws-application-signals-test-framework/actions/runs/8644705184

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[zzhlogin]

I have a general question regarding using container for Github workflow:
The goal of E2E tests is making sure the ADOT Java is working as expected in both EC2 and EKS, which mimic customer behaviours while using ADOT Java. In this PR, we are creating container for workflow runs, does this mean any steps defined within that job will be executed within the context of this container? In this case, we are not running everything from scratch in EC2/EKS, where customer might not do the same. Will this conflict the original goal?

[majanjua-amzn]

I have a general question regarding using container for Github workflow: The goal of E2E tests is making sure the ADOT Java is working as expected in both EC2 and EKS, which mimic customer behaviours while using ADOT Java. In this PR, we are creating container for workflow runs, does this mean any steps defined within that job will be executed within the context of this container? In this case, we are not running everything from scratch in EC2/EKS, where customer might not do the same. Will this conflict the original goal?

@zzhlogin the container Harry is adding here only installs things that we can assume customers will have installed already and without error, for example, git, eksctl, kubectl, awscli, as they would not be able to interact with their EKS cluster or EC2 instance effectively without them. These packages are also not core to what we are trying to test, which is the enablement of Application Signals on these platforms. Checking whether kubectl can be installed is not part of the scope of the test and has only been a blocker/transient issue that stops the actual test from running.

[harrryr]

I have a general question regarding using container for Github workflow: The goal of E2E tests is making sure the ADOT Java is working as expected in both EC2 and EKS, which mimic customer behaviours while using ADOT Java. In this PR, we are creating container for workflow runs, does this mean any steps defined within that job will be executed within the context of this container? In this case, we are not running everything from scratch in EC2/EKS, where customer might not do the same. Will this conflict the original goal?

The customer behavior we are trying to mimic is the act of deploying a sample app and installing app-signals on it. Those are running in the EKS cluster and EC2 environment which are deployed through Terraform. The docker container is for running the github workflow, which isn't part of the customer behavior mimicry but more for just running the E2E tests and it's not something we are expecting the customers to have.

[majanjua-amzn]

Confirmed offline with Harry that a test run with a forced failure still behaves appropriately and sends failure metrics for our alarms.