Skip to content

Commit 81dc490

Browse files
GuptaNavdeep1983npalmnavdeepg2021semantic-release-bot
authored
feat: added publishing to workflow_job event queue for multi runner module. (#2570)
* feat: Remove support check_run (#2521) * chore: Remove support check_run * format, lint * feat: Remove old scale down mechanism (< 0.19.0) (#2519) fix: Remove old cleanup mechanism (< 0.19.0) * chore: Enable pre releases for next branch * feat: Add multi-runner capability (#2472) * feat: Remove support check_run (#2521) * chore: Remove support check_run * format, lint * feat: Remove old scale down mechanism (< 0.19.0) (#2519) fix: Remove old cleanup mechanism (< 0.19.0) * feat: added changes for multi runner. * fix: region. * fix: more fixes. * tuple to list. * fixes. * fixes. * fixes. * fixes. * fixes. * fixes. * fix: formatting. * fix: formatting. * fix: formatting. * fix: moved some blocks outside runner config. * fix: few more updates * fix: liniting. * fix: updated example output * changed runner group name. * fix: updated the tests. * fix: addressed review comments. * fix: linting issues. * fix: formatting. * fix: updated tf version. * fix: Remove removed prerelease option * Add ubuntu runner to example * refactor: use each instead of count * fix: few small issues. * refactor: syncer to count for multi runner * fix: comments. * fix: added Readme. * fix: errors. * move variable to runner config * fix: updated the readme. * Add todos * feat: added windows runner configuration, completed todos and added the weight for runner config matchers. * chore: Update docs * fix: reverted tf versions. * fix: addressed comments. * fix: missed. * fix: formatting. * Update terraform versions in CI * Update terraform versions in CI * Update docs * fix: coverage. * Update docs * improve test coverage webhook * Apply suggestions from code review * fix: formatting. * fix: fixed merge issues. * fix: syntax. Co-authored-by: Niek Palm <[email protected]> Co-authored-by: Niek Palm <[email protected]> Co-authored-by: navdeepg2021 <[email protected]> * chore(ci): Enable ci for next branch BREAKING CHANGE: - Add option to deploy multiple runner at once, see for details: #2472 #2517 - Drop support check_run event, see for details: #2521 #2517 - Remove pre 0.19.0 scale down mechanism, see details: #2519 #2517 * chore(release): 2.0.0-next.1 [skip ci] ## [2.0.0-next.1](philips-labs/terraform-aws-github-runner@v1.12.0...v2.0.0-next.1) (2022-10-19) ### ⚠ BREAKING CHANGES * **ci:** ### Features * Add multi-runner capability ([#2472](https://github.com/philips-labs/terraform-aws-github-runner/issues/2472)) ([c08b335](philips-labs/terraform-aws-github-runner@c08b335)), closes [#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521) [#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519) * Experimental feature - Duplicate workflow job event to extra queue ([#2268](https://github.com/philips-labs/terraform-aws-github-runner/issues/2268)) ([ac046b8](philips-labs/terraform-aws-github-runner@ac046b8)) * Remove old scale down mechanism (< 0.19.0) ([#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519)) ([721d7c3](philips-labs/terraform-aws-github-runner@721d7c3)) * Remove support check_run ([#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521)) ([272a293](philips-labs/terraform-aws-github-runner@272a293)) ### Miscellaneous Chores * **ci:** Enable ci for next branch ([48769ca](philips-labs/terraform-aws-github-runner@48769ca)), closes [#2472](https://github.com/philips-labs/terraform-aws-github-runner/issues/2472) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517) [#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517) [#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517) * chore: fix release workflow * fix: added changes for publishing events to secondary queue. Co-authored-by: Niek Palm <[email protected]> Co-authored-by: Niek Palm <[email protected]> Co-authored-by: navdeepg2021 <[email protected]> Co-authored-by: semantic-release-bot <[email protected]>
1 parent 8a06f30 commit 81dc490

File tree

8 files changed

+51
-46
lines changed

8 files changed

+51
-46
lines changed

‎CHANGELOG.md

Lines changed: 4 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,11 @@
11
# Changelog
22

3-
<<<<<<< HEAD
43
## [1.13.0](https://github.com/philips-labs/terraform-aws-github-runner/compare/v1.12.0...v1.13.0) (2022-10-14)
54

65

76
### Features
87

98
* Experimental feature - Duplicate workflow job event to extra queue ([#2268](https://github.com/philips-labs/terraform-aws-github-runner/issues/2268)) ([985e722](https://github.com/philips-labs/terraform-aws-github-runner/commit/985e722229ce464235d206484df3d989db03e143))
10-
=======
11-
## [2.0.0-next.1](https://github.com/philips-labs/terraform-aws-github-runner/compare/v1.12.0...v2.0.0-next.1) (2022-10-19)
12-
13-
14-
### ⚠ BREAKING CHANGES
15-
16-
* **ci:**
17-
18-
### Features
19-
20-
* Add multi-runner capability ([#2472](https://github.com/philips-labs/terraform-aws-github-runner/issues/2472)) ([c08b335](https://github.com/philips-labs/terraform-aws-github-runner/commit/c08b335ade4e16a89d6e1068d7226c16b1eb1450)), closes [#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521) [#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519)
21-
* Experimental feature - Duplicate workflow job event to extra queue ([#2268](https://github.com/philips-labs/terraform-aws-github-runner/issues/2268)) ([ac046b8](https://github.com/philips-labs/terraform-aws-github-runner/commit/ac046b8eb2a0d2d5e2219ae9ee0023fd8bdf7460))
22-
* Remove old scale down mechanism (< 0.19.0) ([#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519)) ([721d7c3](https://github.com/philips-labs/terraform-aws-github-runner/commit/721d7c3287fd3e0caa57942d0d830b072d90c433))
23-
* Remove support check_run ([#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521)) ([272a293](https://github.com/philips-labs/terraform-aws-github-runner/commit/272a293613d3a1da360f0d61bff92d16ea64216c))
24-
25-
26-
### Miscellaneous Chores
27-
28-
* **ci:** Enable ci for next branch ([48769ca](https://github.com/philips-labs/terraform-aws-github-runner/commit/48769cacc53657ac740537a9dc63d4628b8c2562)), closes [#2472](https://github.com/philips-labs/terraform-aws-github-runner/issues/2472) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517) [#2521](https://github.com/philips-labs/terraform-aws-github-runner/issues/2521) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517) [#2519](https://github.com/philips-labs/terraform-aws-github-runner/issues/2519) [#2517](https://github.com/philips-labs/terraform-aws-github-runner/issues/2517)
29-
>>>>>>> e694884 (chore(release): 2.0.0-next.1 [skip ci])
309

3110
## [1.12.0](https://github.com/philips-labs/terraform-aws-github-runner/compare/v1.11.0...v1.12.0) (2022-10-12)
3211

@@ -729,7 +708,7 @@
729708
- Fix missing permissions for CloudWatch Agent #445 @bennettp123
730709
- Swap scale up/scale down timeout description #468 @jonico
731710
- Fix for invalid configuration #466 @jonico
732-
- Add ssm:GetParameter to runner-ssm-parameters #446 @bennettp123
711+
- Add ssm:GetParameter to runner-ssm-parameters #446 @bennettp123
733712
- Replace crypto #429
734713
- Scale up lambda deprecated attribute #410
735714

@@ -766,17 +745,17 @@ terraform state rm $RESOURCE
766745
### Added
767746

768747
- Allow operator to pass in a list of managed IAM policy ARNs for the runner role #361 @jpalomaki
769-
- expand options for sourcing lambda to include S3 #292 @eky5006
748+
- expand options for sourcing lambda to include S3 #292 @eky5006
770749

771750
## [0.6.0] - 2020-10-10
772751

773752
### Added
774753

775-
- Only allow tagging and termination of runner instances #201 @jpalomaki
754+
- Only allow tagging and termination of runner instances #201 @jpalomaki
776755

777756
### Fixed
778757

779-
- Fix pagination with listing self-hosted runners #202 @HenryNguyen5
758+
- Fix pagination with listing self-hosted runners #202 @HenryNguyen5
780759

781760

782761
## [0.5.0] - 2020-08-25

‎examples/multi-runner/main.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,9 @@ module "multi-runner" {
159159
# runner_binaries_syncer_lambda_zip = "lambdas-download/runner-binaries-syncer.zip"
160160
# runners_lambda_zip = "lambdas-download/runners.zip"
161161

162+
# enable_workflow_job_events_queue = true
162163
# override delay of events in seconds
163164

164165
# log_level = "debug"
166+
165167
}

‎modules/multi-runner/queues.tf

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,3 +70,22 @@ resource "aws_sqs_queue_policy" "build_queue_dlq_policy" {
7070
queue_url = aws_sqs_queue.queued_builds_dlq[each.key].id
7171
policy = data.aws_iam_policy_document.deny_unsecure_transport.json
7272
}
73+
74+
resource "aws_sqs_queue" "webhook_events_workflow_job_queue" {
75+
count = var.enable_workflow_job_events_queue ? 1 : 0
76+
name = "${var.prefix}-webhook_events_workflow_job_queue"
77+
delay_seconds = var.workflow_job_queue_configuration.delay_seconds
78+
visibility_timeout_seconds = var.workflow_job_queue_configuration.visibility_timeout_seconds
79+
message_retention_seconds = var.workflow_job_queue_configuration.message_retention_seconds
80+
fifo_queue = false
81+
receive_wait_time_seconds = 0
82+
content_based_deduplication = false
83+
redrive_policy = null
84+
85+
sqs_managed_sse_enabled = var.queue_encryption.sqs_managed_sse_enabled
86+
kms_master_key_id = var.queue_encryption.kms_master_key_id
87+
kms_data_key_reuse_period_seconds = var.queue_encryption.kms_data_key_reuse_period_seconds
88+
89+
tags = var.tags
90+
}
91+

‎modules/multi-runner/variables.tf

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -489,3 +489,24 @@ variable "pool_lambda_reserved_concurrent_executions" {
489489
type = number
490490
default = 1
491491
}
492+
493+
variable "enable_workflow_job_events_queue" {
494+
description = "Enabling this experimental feature will create a secondory sqs queue to wich a copy of the workflow_job event will be delivered."
495+
type = bool
496+
default = false
497+
}
498+
499+
variable "workflow_job_queue_configuration" {
500+
description = "Configuration options for workflow job queue which is only applicable if the flag enable_workflow_job_events_queue is set to true."
501+
type = object({
502+
delay_seconds = number
503+
visibility_timeout_seconds = number
504+
message_retention_seconds = number
505+
})
506+
default = {
507+
"delay_seconds" : null,
508+
"visibility_timeout_seconds" : null,
509+
"message_retention_seconds" : null
510+
}
511+
}
512+

‎modules/multi-runner/webhook.tf

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ module "webhook" {
66

77
runner_config = local.runner_config
88
github_app_webhook_secret_arn = module.ssm.parameters.github_app_webhook_secret.arn
9+
sqs_workflow_job_queue = length(aws_sqs_queue.webhook_events_workflow_job_queue) > 0 ? aws_sqs_queue.webhook_events_workflow_job_queue[0] : null
10+
911

1012
lambda_s3_bucket = var.lambda_s3_bucket
1113
webhook_lambda_s3_key = var.webhook_lambda_s3_key

‎modules/runner-binaries-syncer/lambdas/runner-binaries-syncer/yarn.lock

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,7 @@
358358
resolved "https://registry.yarnpkg.com/@babel/parser/-/parser-7.18.9.tgz#f2dde0c682ccc264a9a8595efd030a5cc8fd2539"
359359
integrity sha512-9uJveS9eY9DJ0t64YbIBZICtJy8a5QrDEVdiLCG97fVLpDTpGX7t8mMSb6OWw6Lrnjqj4O8zwjELX3dhoMgiBg==
360360

361+
361362
"@babel/parser@^7.1.0", "@babel/parser@^7.14.7", "@babel/parser@^7.16.0", "@babel/parser@^7.16.4", "@babel/parser@^7.16.7", "@babel/parser@^7.17.3", "@babel/parser@^7.17.8", "@babel/parser@^7.18.10", "@babel/parser@^7.18.6", "@babel/parser@^7.19.3", "@babel/parser@^7.19.4":
362363
version "7.20.0"
363364
resolved "https://registry.yarnpkg.com/@babel/parser/-/parser-7.20.0.tgz#b26133c888da4d79b0d3edcf42677bcadc783046"

‎modules/runners/lambdas/runners/yarn.lock

Lines changed: 0 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3704,29 +3704,10 @@ jest-snapshot@^29.2.1:
37043704
pretty-format "^29.2.1"
37053705
semver "^7.3.5"
37063706

3707-
<<<<<<< HEAD
3708-
jest-util@^29.0.0, jest-util@^29.2.1:
3709-
version "29.2.1"
3710-
resolved "https://registry.yarnpkg.com/jest-util/-/jest-util-29.2.1.tgz#f26872ba0dc8cbefaba32c34f98935f6cf5fc747"
3711-
integrity sha512-P5VWDj25r7kj7kl4pN2rG/RN2c1TLfYYYZYULnS/35nFDjBai+hBeo3MDrYZS7p6IoY3YHZnt2vq4L6mKnLk0g==
3712-
=======
3713-
jest-util@^29.0.0, jest-util@^29.2.0:
3714-
version "29.2.0"
3715-
resolved "https://registry.yarnpkg.com/jest-util/-/jest-util-29.2.0.tgz#797935697e83a5722aeba401ed6cd01264295566"
3716-
integrity sha512-8M1dx12ujkBbnhwytrezWY0Ut79hbflwodE+qZKjxSRz5qt4xDp6dQQJaOCFvCmE0QJqp9KyEK33lpPNjnhevw==
3717-
dependencies:
3718-
"@jest/types" "^29.2.0"
3719-
"@types/node" "*"
3720-
chalk "^4.0.0"
3721-
ci-info "^3.2.0"
3722-
graceful-fs "^4.2.9"
3723-
picomatch "^2.2.3"
3724-
37253707
jest-util@^29.0.0, jest-util@^29.1.2:
37263708
version "29.1.2"
37273709
resolved "https://registry.yarnpkg.com/jest-util/-/jest-util-29.1.2.tgz#ac5798e93cb6a6703084e194cfa0898d66126df1"
37283710
integrity sha512-vPCk9F353i0Ymx3WQq3+a4lZ07NXu9Ca8wya6o4Fe4/aO1e1awMMprZ3woPFpKwghEOW+UXgd15vVotuNN9ONQ==
3729-
>>>>>>> 5e17d6f (feat: Add multi-runner capability (#2472))
37303711
dependencies:
37313712
"@jest/types" "^29.2.1"
37323713
"@types/node" "*"

‎modules/webhook/lambdas/webhook/src/webhook/handler.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,10 +63,10 @@ export async function handle(headers: IncomingHttpHeaders, body: string): Promis
6363

6464
const workflowJobEvent = payload as WorkflowJobEvent;
6565
response = await handleWorkflowJob(workflowJobEvent, githubEvent, queuesConfig);
66-
await sendWorkflowJobEvents(githubEvent, workflowJobEvent);
66+
await sendWorkflowJobEvents(workflowJobEvent);
6767
return response;
6868
}
69-
async function sendWorkflowJobEvents(githubEvent: string, workflowEventPayload: WorkflowJobEvent) {
69+
async function sendWorkflowJobEvents(workflowEventPayload: WorkflowJobEvent) {
7070
await sendWebhookEventToWorkflowJobQueue({
7171
workflowJobEvent: workflowEventPayload,
7272
});

0 commit comments

Comments
 (0)