Storing credentials in the profile

[ms32035]

Is your feature request related to a problem? Please describe.
I'm trying to store some sensitive variables like storage account credentials in the dbt profile, and then use them in sql using target. So far the only feasible solution seems to be using custom parameters like log_comment custom_settings: {'log_comment': 'myaccountkey=='} because others are not passed to the target

Describe the solution you'd like
extend the profile to allow additional credentials, for example

extra_params:
  var1: val1

and pass that to the target

Describe alternatives you've considered
setting env variables and referring to them in a macro isn't acceptable. the value must come from a profile

[babaMar]

@ms32035 can't you not use env_var macro in the profile?

[ms32035]

@babaMar the problem is not where I take it from, but where I put it so that it's available in clickhouse.
Could you expand on what do you exactly suggest?

[morsapaes]

It'd be great if you could be more specific and, ideally, provide a minimal mock of what you're hoping to do. Is the goal to pass sensitive credentials for external storage? Or is it for table engines?

The dbt profile only supports connection configurations for the ClickHouse connection. It shouldn't support connection configurations for other external systems. If you need these credentials at the project- or model- level, the implementation would look more like this, which allows you to inject the credentials in the model config or dbt_profile.yml. You can then use Jinja to conditionally pass different credentials based on the target profile. The profile is reserved for client-level options.

[ms32035]

@morsapaes we use Airflow and Cosmos to run this dbt connector. Specifically, Cosmos uses this mapping https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/profiles/clickhouse/user_pass.py to map attributes from an Airflow connection to a dbt profile.
It's possible to extend this to map Airflow extras, onto any custom attributes in the clickhouse dbt profile. In our case we use that to store credentials for an external storage that Clickhouse should read from, hence the credenials are profile, not project or model level.