Skip to content

[image-builder-mk3] log errors for auth #18611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 30, 2023
Merged

Conversation

kylos101
Copy link
Contributor

@kylos101 kylos101 commented Aug 28, 2023

Description

This will help us troubleshoot:

  • credential reload via watch
  • potential ECR authN issues
  • potential additionalAuth issues
Summary generated by Copilot

🤖 Generated by Copilot at 8082e76

Improve error logging in image-builder-mk3 auth package. Add log messages with error and file or image name in auth.go.

Related Issue(s)

Relates to ENG-757

How to test

  1. Assert you have a kubectx for your preview env
  2. Backup the existing pull secret: kubectl get secrets gcp-sa-registry-auth -o yaml > /tmp/secret-backup.yaml
  3. Setup some test data, you can use the original secret, FAKE_VALID_TOKEN, or FAKE_INVALID_TOKEN (below):
set -x

# passes docker cli validation, but cannot actually log in
export FAKE_VALID_TOKEN=$(cat << 'EOF'
_json_key:{
  "type": "service_account",
  "project_id": "123",
  "private_key_id": "0",
  "private_key": "-----BEGIN PRIVATE KEY-----\0\n-----END PRIVATE KEY-----\n",
  "client_email": "foo.com",
  "client_id": "0",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/foo"
}
EOF
)

# fails docker cli validation
export FAKE_INVALID_TOKEN=$(echo -n "foo" | base64 -w0)

export AUTH_TOKEN=$(echo -n ${FAKE_INVALID_TOKEN} | base64 -w0)

cat << EOF > config.json
{
    "auths": {
        "eu.gcr.io": {
            "auth": "${AUTH_TOKEN}"
        },
        "europe-docker.pkg.dev": {
            "auth": "${AUTH_TOKEN}"
        }
    }
}
EOF

kubectl create secret generic gcp-sa-registry-auth \
    --from-file=.dockerconfigjson=config.json \
    --type=kubernetes.io/dockerconfigjson \
    -o yaml --dry-run=client | kubectl apply --server-side --force-conflicts -f -
  1. Follow the logs to observe the watch and errors: kubectl logs --follow deployment/image-builder-mk3
  2. Reset the secret back from the backup: kubectl apply --server-side --force-conflicts -f /tmp/secret-backup.yaml

Test Scenarios

Success (restored from the backup):

2023-08-29T20:27:02.17122308Z  INFO "reloading file after change"
2023-08-29T20:27:02.171413799Z  INFO "reloading auth from Docker config"
2023-08-29T20:27:02.17151425Z  INFO "file has changed: /config/pull-secret/pull-secret.json"

A token that has invalid format, FAKE_INVALID_TOKEN (this error is from docker cli here):

2023-08-29T20:29:33.114833864Z  INFO "reloading file after change"
2023-08-29T20:29:33.114961164Z  INFO "reloading auth from Docker config"
2023-08-29T20:29:33.115032464Z  ERROR "failed loading from file"
error loading Docker config from /config/pull-secret/pull-secret.json: Invalid auth configuration file

A token that appears valid, FAKE_VALID_TOKEN (passed field validation, contains junk):

2023-08-29T21:19:02.915601783Z  INFO "reloading file after change"
2023-08-29T21:19:02.915748863Z  INFO "reloading auth from Docker config"
2023-08-29T21:19:02.915811433Z  INFO "file has changed: /config/pull-secret/pull-secret.json"

Documentation

Preview status

Gitpod was successfully deployed to your preview environment.

Build Options

Build
  • /werft with-werft
    Run the build with werft instead of GHA
  • leeway-no-cache
  • /werft no-test
    Run Leeway with --dont-test
Publish
  • /werft publish-to-npm
  • /werft publish-to-jb-marketplace
Installer
  • analytics=segment
  • with-dedicated-emulation
  • workspace-feature-flags
    Add desired feature flags to the end of the line above, space separated
Preview Environment / Integration Tests
  • /werft with-local-preview
    If enabled this will build install/preview
  • /werft with-preview
  • /werft with-large-vm
  • /werft with-gce-vm
    If enabled this will create the environment on GCE infra
  • with-integration-tests=all
    Valid options are all, workspace, webapp, ide, jetbrains, vscode, ssh. If enabled, with-preview and with-large-vm will be enabled.
  • with-monitoring

/hold

This will help us troubleshoot:
* credential reload via watch
* potential ECR authN issues
* potential additionalAuth issues
@kylos101 kylos101 force-pushed the kylos101/add-auth-err-logs branch from f7fb540 to 8c5ee96 Compare August 29, 2023 21:02
Copy link
Contributor Author

@kylos101 kylos101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to share comments 🙃

Comment on lines +163 to +168
defer func() {
if err != nil {
err = fmt.Errorf("error with ECR authenticate: %w", err)
log.WithError(err).WithField("registry", registry).Error("failed ECR authenticate")
}
}()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few errors in func (ath *ECRAuthenticator) Authenticate(ctx context.Context, registry string), which this could help surface.

}

segs := strings.Split(ath.ecrAuth, ":")
if len(segs) != 2 {
return nil, fmt.Errorf("cannot understand ECR token. Expected 2 segments, got %d", len(segs))
err = fmt.Errorf("cannot understand ECR token. Expected 2 segments, got %d", len(segs))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By setting err here, it's value will be included in the above defer.

@roboquat roboquat merged commit e754fc6 into main Aug 30, 2023
@roboquat roboquat deleted the kylos101/add-auth-err-logs branch August 30, 2023 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants