Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.

add --state flag feature to dbt integration #600

Merged
merged 6 commits into from
Jun 14, 2023
Merged

add --state flag feature to dbt integration #600

merged 6 commits into from
Jun 14, 2023

Conversation

dlawin
Copy link
Contributor

@dlawin dlawin commented Jun 5, 2023

Adds a new flag --state for use with the dbt integration

A user can use this flag to specify the "production" manifest location e.g.:
data-diff --dbt --state ~/prod_manifest.json

This removes the need for configuration in the dbt_project.yml, and covers scenarios not covered via configuration. The prod manifest is used as the source of truth for production model locations vs. via configuration.

Changes:

  • added prod_manifest_obj to DbtParser
  • renamed manifest_obj in DbtParser to dev_manifest_obj to differentiate
  • Renamed DbtParser.get_datadiff_variables to DbtParser.get_datadiff_config (this was easily confused with _get_diff_vars)
    • Altered to return a TDatadiffConfig object vs. the previous dictionary
  • get_manifest_obj altered to be generic (can be used for any path)
  • _get_diff_vars altered to construct prod path based the production manifest when it is present
    • new method for manifest _get_prod_path_from_manifest
    • moved existing config functionality to new method _get_prod_path_from_config
  • DataDiffDbtProjectVarsNotFoundError now thrown when config is not present AND not using --state

Resolves #454

@dlawin dlawin added enhancement New feature or request --dbt Issues/features related to the dbt integration labels Jun 5, 2023
@dlawin dlawin self-assigned this Jun 5, 2023
@dlawin dlawin merged commit 0d624fa into master Jun 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
--dbt Issues/features related to the dbt integration enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to use a manifest for prod model paths (--state)
1 participant