Skip to content

[management] Add postgres support for activity event store #3890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 4, 2025

Conversation

bcmmbaga
Copy link
Contributor

@bcmmbaga bcmmbaga commented May 28, 2025

Describe your changes

Adds postgres as an optional storage engine for activity events while keeping sqlite as the default. To use postgres, set the environment variable NB_ACTIVITY_EVENT_STORE_ENGINE=postgres and provide a connection string via NB_ACTIVITY_EVENT_POSTGRES_DSN.

Migrating existing events to Postgres

  1. Back up the current sqlite store
mkdir backup
docker compose cp -a management:/var/lib/netbird/. backup/
  1. Load the data into Postgres
pgloader --type sqlite backup/events.db "postgresql://<PG_USER>:<PG_PASSWORD>@<PG_HOST>:<PG_PORT>/<PG_DB_NAME>"

Issue ticket number and link

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • Extended the README / documentation, if necessary

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

@bcmmbaga bcmmbaga marked this pull request as ready for review May 29, 2025 11:48
@Copilot Copilot AI review requested due to automatic review settings May 29, 2025 11:48
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds optional Postgres support for the activity event store by extracting database initialization and pooling logic, while keeping SQLite as the default engine.

  • Extracts database setup into initDatabase/configureConnectionPool to support multiple engines
  • Introduces NB_ACTIVITY_EVENT_STORE_ENGINE, NB_ACTIVITY_EVENT_POSTGRES_DSN, and NB_SQL_MAX_OPEN_CONNS environment variables
  • Adds Postgres driver import and connection logic
Comments suppressed due to low confidence (3)

management/server/activity/sqlite/sqlite.go:46

  • [nitpick] The function name NewSQLiteStore no longer accurately reflects its support for Postgres. Consider renaming to NewEventStore or NewStore for clarity.
func NewSQLiteStore(ctx context.Context, dataDir string, encryptionKey string) (

management/server/activity/sqlite/sqlite.go:31

  • The new environment variables (NB_ACTIVITY_EVENT_STORE_ENGINE, NB_ACTIVITY_EVENT_POSTGRES_DSN, NB_SQL_MAX_OPEN_CONNS) should be documented in the repo README or a configuration guide so users know how to configure Postgres mode.
const (

management/server/activity/sqlite/sqlite.go:240

  • We should add unit tests for the Postgres initialization path and the error path when NB_ACTIVITY_EVENT_POSTGRES_DSN is unset to ensure coverage of the new logic.
func initDatabase(ctx context.Context, dataDir string) (*gorm.DB, error) {

@bcmmbaga bcmmbaga force-pushed the activity-events-postgres-support branch from 50e1f62 to 7a9c80a Compare May 29, 2025 12:21
Signed-off-by: bcmmbaga <[email protected]>
@pnmcosta
Copy link
Contributor

pnmcosta commented May 30, 2025

@bcmmbaga there's a problem with the migration migrateDuplicateDeletedUsers that relies on sqlite's rowid.

Once I migrated my local env to use postgres, I started having Error: failed to initialize database: events database migration: ERROR: column "rowid" does not exist (SQLSTATE 42703) when starting management.

I suspect there's something up with gorm's primary key check with postgres or the schema the delete users table was created with. I started management once before executing pgloader to have gorm migrate the tables.

@bcmmbaga bcmmbaga force-pushed the activity-events-postgres-support branch from 7ccb87b to 5024a08 Compare May 30, 2025 09:58
Copy link

@bcmmbaga bcmmbaga merged commit b604c66 into main Jun 4, 2025
38 of 39 checks passed
@bcmmbaga bcmmbaga deleted the activity-events-postgres-support branch June 4, 2025 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants