Skip to content

Commit 2956a28

Browse files
millintomasfarias
millin
and
tomasfarias
authored
feat: Support dbt 1.9 (#143)
* Support dbt 1.9 * Remove DBT_INSTALLED_GTE_1_8 * Update * Fix test issue * Update actions/upload-artifact & actions/download-artifact * Fix typing * Fix artifacts * Update CI * Fix docs build * Merge artifacts * fix: Update docs dependencies * fix: Pin zipp version * fix: Update readthedocs.yaml * chore: Remove unused requirements.txt --------- Co-authored-by: Tomás Farías Santana <[email protected]>
1 parent 8859071 commit 2956a28

File tree

16 files changed

+4306
-3518
lines changed

16 files changed

+4306
-3518
lines changed

‎.github/workflows/ci.yaml‎

Lines changed: 26 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -26,26 +26,22 @@ jobs:
2626
- '3.11'
2727
- '3.10'
2828
- '3.9'
29-
- '3.8'
3029
airflow-version:
31-
- '2.9.2'
32-
- '2.8.4'
33-
- '2.7.3'
30+
- '2.10'
31+
- '2.9'
32+
- '2.8'
3433
dbt-version:
34+
- 1.9
3535
- 1.8
36-
- 1.7
3736
exclude:
3837
# Incompatible combinations
3938
- python-version: 3.12
40-
airflow-version: '2.8.4'
41-
42-
- python-version: 3.12
43-
airflow-version: '2.7.3'
39+
airflow-version: '2.8'
4440

4541
runs-on: ubuntu-latest
4642
steps:
4743
- name: Harden Runner
48-
uses: step-security/harden-runner@v2.8.1
44+
uses: step-security/harden-runner@v2.10.4
4945
with:
5046
egress-policy: block
5147
allowed-endpoints: >
@@ -54,6 +50,7 @@ jobs:
5450
hub.getdbt.com:443
5551
github.com:80
5652
github.com:443
53+
gitlab.com:22
5754
gitlab.com:80
5855
gitlab.com:443
5956
objects.githubusercontent.com:443
@@ -64,28 +61,29 @@ jobs:
6461
esm.ubuntu.com:443
6562
motd.ubuntu.com:443
6663
packages.microsoft.com:80
64+
packages.microsoft.com:443
6765
ppa.launchpadcontent.net:443
6866
security.ubuntu.com:80
6967
7068
- run: |
7169
sudo apt-get update
7270
sudo apt-get install --yes --no-install-recommends postgresql
7371
74-
- uses: actions/checkout@v4.1.7
72+
- uses: actions/checkout@v4.2.2
7573
- name: Set up Python ${{ matrix.python-version }}
76-
uses: actions/setup-python@v5.1.0
74+
uses: actions/setup-python@v5.4.0
7775
with:
7876
python-version: ${{ matrix.python-version }}
7977

8078
- name: Install Poetry
81-
uses: abatilo/actions-poetry@v2.3.0
79+
uses: abatilo/actions-poetry@v2.4.0
8280
with:
83-
poetry-version: 1.8.3
81+
poetry-version: 2.0.1
8482

8583
- name: Install Airflow & dbt
8684
run: |
8785
poetry env use ${{ matrix.python-version }}
88-
poetry add "apache-airflow==${{ matrix.airflow-version }}" \
86+
poetry add "apache-airflow~=${{ matrix.airflow-version }}.0" \
8987
"dbt-core~=${{ matrix.dbt-version }}.0" \
9088
"dbt-postgres~=${{ matrix.dbt-version }}.0" \
9189
--python ${{ matrix.python-version }}
@@ -99,14 +97,14 @@ jobs:
9997
- name: Static type checking with mypy
10098
# We only run mypy on the latest supported versions of Airflow & dbt,
10199
# so it is currently impossible to write conditions for that depend on package versions.
102-
if: matrix.python-version == '3.12' && matrix.airflow-version == '2.9.2' && matrix.dbt-version == '1.8'
100+
if: matrix.python-version == '3.12' && matrix.airflow-version == '2.10' && matrix.dbt-version == '1.9'
103101
run: poetry run mypy .
104102

105103
- name: Code formatting with black
106104
run: poetry run black --check .
107105

108106
- name: Set COVERAGE_FILE in environment
109-
run: echo "COVERAGE_FILE=.coverage.${{ matrix.python-version }}.${{ matrix.airflow-version }}" >> $GITHUB_ENV
107+
run: echo "COVERAGE_FILE=.coverage.${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ matrix.dbt-version }}" >> $GITHUB_ENV
110108

111109
- name: Run tests with pytest
112110
run: poetry run coverage run -m pytest -v tests/ airflow_dbt_python/utils/
@@ -117,11 +115,11 @@ jobs:
117115
GITHUB_USERNAME: ${{ secrets.GH_USERNAME }}
118116

119117
- name: Upload code coverage
120-
uses: actions/upload-artifact@v3
118+
uses: actions/upload-artifact@v4
121119
with:
122-
name: coverage-data
120+
name: coverage-data-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ matrix.dbt-version }}
123121
path: ".coverage.*"
124-
if-no-files-found: ignore
122+
include-hidden-files: true
125123

126124
coverage:
127125
name: Combine and check coverage
@@ -130,7 +128,7 @@ jobs:
130128

131129
steps:
132130
- name: Harden Runner
133-
uses: step-security/harden-runner@v2.8.1
131+
uses: step-security/harden-runner@v2.10.4
134132
with:
135133
egress-policy: block
136134
allowed-endpoints: >
@@ -139,23 +137,23 @@ jobs:
139137
api.github.com:443
140138
pypi.org:443
141139
142-
- uses: actions/checkout@v4.1.7
143-
- uses: actions/setup-python@v5.1.0
140+
- uses: actions/checkout@v4.2.2
141+
- uses: actions/setup-python@v5.4.0
144142
with:
145143
python-version: '3.12'
146144

147145
- name: Install Poetry
148-
uses: abatilo/actions-poetry@v2.3.0
146+
uses: abatilo/actions-poetry@v2.4.0
149147
with:
150-
poetry-version: 1.8.3
148+
poetry-version: 2.0.1
151149

152150
- name: Install airflow-dbt-python with Poetry
153151
run: poetry install --with dev -E airflow-providers
154152

155153
- name: Download coverage data.
156-
uses: actions/download-artifact@v3
154+
uses: actions/download-artifact@v4
157155
with:
158-
name: coverage-data
156+
merge-multiple: true
159157

160158
- name: Combine coverage & fail if it's <95%.
161159
run: |
@@ -171,7 +169,7 @@ jobs:
171169
poetry run coverage report --fail-under=95
172170
173171
- name: Upload HTML report if check failed.
174-
uses: actions/upload-artifact@v3
172+
uses: actions/upload-artifact@v4
175173
if: ${{ failure() }}
176174
with:
177175
name: html-report

‎.github/workflows/docs_pages.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,15 @@ jobs:
1313
with:
1414
fetch-depth: 0
1515

16-
- name: Set up Python '3.10'
17-
uses: actions/setup-python@v2
16+
- name: Set up Python '3.12'
17+
uses: actions/setup-python@v5.4.0
1818
with:
19-
python-version: '3.10'
19+
python-version: '3.12'
2020

2121
- name: Install Poetry
22-
uses: abatilo/actions-poetry@v2.1.4
22+
uses: abatilo/actions-poetry@v2.4.0
2323
with:
24-
poetry-version: 1.3.2
24+
poetry-version: 2.0.1
2525

2626
- name: Install airflow-dbt-python with Poetry
2727
run: poetry install -E airflow-providers --with docs

‎.github/workflows/pypi_deploy.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,15 @@ jobs:
99
steps:
1010
- uses: actions/checkout@v2
1111

12-
- name: Set up Python 3.10
13-
uses: actions/setup-python@v2
12+
- name: Set up Python 3.12
13+
uses: actions/setup-python@v5.4.0
1414
with:
15-
python-version: '3.10'
15+
python-version: '3.12'
1616

1717
- name: Install Poetry
18-
uses: abatilo/actions-poetry@v2.1.4
18+
uses: abatilo/actions-poetry@v2.4.0
1919
with:
20-
poetry-version: 1.3.2
20+
poetry-version: 2.0.1
2121

2222
- name: Install airflow-dbt-python with Poetry
2323
run: poetry install

‎.github/workflows/tagged_release.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,14 @@ jobs:
2121
uses: actions/[email protected]
2222

2323
- name: Set up Python 3.10
24-
uses: actions/setup-python@v2
24+
uses: actions/setup-python@v5.4.0
2525
with:
2626
python-version: '3.10'
2727

2828
- name: Install Poetry
29-
uses: abatilo/actions-poetry@v2.1.4
29+
uses: abatilo/actions-poetry@v2.4.0
3030
with:
31-
poetry-version: 1.3.2
31+
poetry-version: 2.0.1
3232

3333
- name: Install airflow-dbt-python with Poetry
3434
run: poetry install

‎.readthedocs.yaml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,22 @@
11
version: 2
22

33
build:
4-
os: "ubuntu-22.04"
4+
os: "ubuntu-lts-latest"
55
tools:
6-
python: "3.11"
6+
python: "3.12"
77
apt_packages:
88
- graphviz
99
jobs:
10-
post_install:
11-
- pip install poetry==1.3.2
12-
- poetry config virtualenvs.create false
10+
pre_install:
11+
- python -m pip install poetry==2.0.1
12+
- poetry config virtualenvs.create false --local
1313
- poetry install -E git -E s3 --with docs
14+
- poetry export -E git -E s3 --with docs -f requirements.txt --without-hashes --output docs/requirements.txt
15+
16+
python:
17+
install:
18+
- requirements: docs/requirements.txt
19+
20+
sphinx:
21+
builder: dirhtml
22+
configuration: docs/conf.py

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples
1515
## Requirements
1616

1717
Before using *airflow-dbt-python*, ensure you meet the following requirements:
18-
* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.7.5 or later.
19-
* An Airflow environment using version 2.7 or later.
18+
* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.8 or later.
19+
* An Airflow environment using version 2.8 or later.
2020

2121
* If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow.
2222
* If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).

‎airflow_dbt_python/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@
33
__author__ = "Tomás Farías Santana"
44
__copyright__ = "Copyright 2021 Tomás Farías Santana"
55
__title__ = "airflow-dbt-python"
6-
__version__ = "2.1.0"
6+
__version__ = "2.2.0"

‎airflow_dbt_python/hooks/dbt.py‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@
2525
from airflow.hooks.base import BaseHook
2626
from airflow.models.connection import Connection
2727

28-
from airflow_dbt_python.utils.version import DBT_INSTALLED_GTE_1_8
29-
3028
if sys.version_info >= (3, 11):
3129
from contextlib import chdir as chdir_ctx
3230
else:
@@ -400,13 +398,7 @@ def setup_dbt_logging(self, task: BaseTask, debug: Optional[bool]):
400398
default_stdout. As these are initialized by the CLI app, we need to
401399
initialize them here.
402400
"""
403-
if DBT_INSTALLED_GTE_1_8:
404-
from dbt.events.logging import setup_event_logger
405-
else:
406-
from dbt.events.functions import ( # type: ignore[no-redef]
407-
setup_event_logger,
408-
)
409-
401+
from dbt.events.logging import setup_event_logger
410402
from dbt.flags import get_flags
411403

412404
flags = get_flags()

0 commit comments

Comments
 (0)