Use the same logic as dbt-core for the path for the project and profiles

[dbeatty10]

resolves #416

Overview

The changes in this PR allow using the same logic as dbt. The precedence for the location of profiles.yml changed starting dbt-core>=1.4.0.

This PR is designed to determine the version of dbt indicated within the manifest and use the applicable logic.

Behavior for dbt >= 1.3

The parent directory for profiles.yml is determined using the following precedence:

1. --profiles-dir option (--dbt-profiles-dir in data-diff)
2. DBT_PROFILES_DIR environment variable
3. current working directory ← NEW
4. ~/.dbt/ directory

Behavior prior to 1.3:

The parent directory for profiles.yml is determined using the following precedence:

1. --profiles-dir option (--dbt-profiles-dir in data-diff)
2. DBT_PROFILES_DIR environment variable
3. ~/.dbt/ directory

Both are documented here. The version needs to be changed at the top of the screen to switch between versions.

Implementation details

Path is used liberally.

Source of the logic

The relevant logic is located here in dbt-core:

• https://github.com/dbt-labs/dbt-core/blob/c952d44ec5c2506995fbad75320acbae49125d3d/core/dbt/cli/resolvers.py#L6-L13

[dbeatty10]

@erezsh I made an attempt to update the mocks so that they are Paths rather than strs.

If someone is willing to re-run the CI workflows, we can see if that worked or not.

[dbeatty10]

I'm getting this error, so I'll just need to look at the way the test is expressed and update the test accordingly:

Expected: open('target/run_results.json')
Actual: open(PosixPath('target/run_results.json'))

Alternatively, I could cast all the Paths to strs and the tests should pass. But it seems like using a Path in open should be acceptable.

[dlawin]

I'm getting this error, so I'll just need to look at the way the test is expressed and update the test accordingly:

Expected: open('target/run_results.json')
Actual: open(PosixPath('target/run_results.json'))

Alternatively, I could cast all the Paths to strs and the tests should pass. But it seems like using a Path in open should be acceptable.

yeah I should have used Path originally

[dlawin]

This is dbt's behavior only as of >=1.4 correct?

Just worried we cause a disconnect for dbt < 1.4

[dlawin]

This is dbt's behavior only as of >=1.4 correct?

Just worried we cause a disconnect for dbt < 1.4

I think we should make this change and align with dbt, but I do think we should also consider how to handle <1.4

The shift of this default location was a little jarring for dbt users, and anecdotally I've seen many work around the change via the environment variable export DBT_PROFILES_DIR=~/.dbt/

If we had a data-diff --dbt-debug feature analogous to dbt debug, or were a little more verbose when starting a data-diff --dbt, I'd feel a little more comfortable about it.

The location of this file is a little chaotic at the moment, and the <dbt_project>/profiles.yml file will often have {{env_var()}}s that are not set on local dev environments. With --dbt's current state, there could be some very obscure errors. Ironically, I think those on 1.4+ are more aware of this location being a potential issue than those who aren't

[dbeatty10]

Agreed that it makes sense to align with the behavior based on the dbt version.

I just pushed some changes to try handling both the DBT_PROFILES_DIR environment variable as well as the change in behavior between dbt 1.2 and dbt 1.3.

⚠️ I did not test them locally yet though.

Behavior for dbt >= 1.3

The parent directory for profiles.yml is determined using the following precedence:

1. --profiles-dir option
2. DBT_PROFILES_DIR environment variable
3. current working directory
4. ~/.dbt/ directory

Behavior prior to 1.3:

The parent directory for profiles.yml is determined using the following precedence:

1. --profiles-dir option
2. DBT_PROFILES_DIR environment variable
3. ~/.dbt/ directory

Both are documented here. The version needs to be changed at the top of the screen to switch between versions.

[dlawin]

Works great, really appreciate the contributions Doug!

[dbeatty10]

Like the wording update @dlawin -- thanks for putting that in there 🏅

[williebsweet]

Thanks, @dbeatty10!