Skip to content

Commit 150eba2

Browse files
committed
Update Doris documentation and engine adapter
1 parent 2b4ccc8 commit 150eba2

File tree

2 files changed

+16
-29
lines changed

2 files changed

+16
-29
lines changed

docs/integrations/engines/doris.md

Lines changed: 6 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,7 @@ MODEL (
8787

8888
## Table Properties
8989

90-
The Doris adapter supports a comprehensive set of table properties that can be configured in the `physical_properties` section of your model. These properties are processed by the `_build_table_properties_exp` method.
90+
The Doris adapter supports a comprehensive set of table properties that can be configured in the `physical_properties` section of your model.
9191

9292
### Core Table Properties
9393

@@ -96,7 +96,7 @@ The Doris adapter supports a comprehensive set of table properties that can be c
9696
| `unique_key` | `Tuple[str]` or `str` | Defines unique key columns for UNIQUE model | `('user_id')` or `'user_id'` |
9797
| `duplicate_key` | `Tuple[str]` or `str` | Defines key columns for DUPLICATE model | `('user_id', 'event_time')` |
9898
| `distributed_by` | `Dict` | Distribution configuration | See Distribution section |
99-
| `partitioned_by_expr` | `Tuple[str]` or `str` | Custom partition expression | `'FROM ("2000-11-14") TO ("2099-11-14") INTERVAL 1 MONTH'` |
99+
| `partitions` | `Tuple[str]` or `str` | Custom partition expression | `'FROM ("2000-11-14") TO ("2099-11-14") INTERVAL 1 MONTH'` |
100100

101101
### Distribution Configuration
102102

@@ -139,16 +139,16 @@ MODEL (
139139

140140
### Partitioning
141141

142-
Doris supports range partitioning to improve query performance. SQLMesh automatically translates partitioning into Doris's `PARTITION BY RANGE` syntax.
142+
Doris supports range partitioning and list partitioning to improve query performance.
143143

144144
**Custom Partition Expression:**
145145
```sql
146146
MODEL (
147147
name my_partitioned_model,
148148
kind INCREMENTAL_BY_TIME_RANGE(time_column (event_date, '%Y-%m-%d')),
149-
partitioned_by event_date
149+
partitioned_by RANGE(event_date),
150150
physical_properties (
151-
partitioned_by_expr = 'FROM ("2000-11-14") TO ("2099-11-14") INTERVAL 2 YEAR',
151+
partitions = 'FROM ("2000-11-14") TO ("2099-11-14") INTERVAL 2 YEAR',
152152
),
153153
);
154154
```
@@ -157,7 +157,7 @@ MODEL (
157157
MODEL (
158158
name my_custom_partitioned_model,
159159
kind FULL,
160-
partitioned_by event_date,
160+
partitioned_by RANGE(event_date),
161161
physical_properties (
162162
partitioned_by_expr = (
163163
'PARTITION `p2023` VALUES [("2023-01-01"), ("2024-01-01"))',
@@ -169,11 +169,6 @@ MODEL (
169169
);
170170
```
171171

172-
**Important Notes:**
173-
- For UNIQUE KEY tables, partition columns must be included in the `unique_key` definition
174-
- If partition columns are not in the unique key, partitioning will be skipped with a warning
175-
- The adapter automatically handles partition expression generation for time-based partitioning
176-
177172
### Generic Properties
178173

179174
Any additional properties in `physical_properties` are passed through as Doris table properties:
@@ -261,7 +256,6 @@ MODEL (
261256
| `refresh_trigger` | Schedule for automatic refresh | `'MANUAL'`, `'ON SCHEDULE INTERVAL 1 HOUR'`, `'ON COMMIT'` |
262257
| `unique_key` | Unique key columns | `'user_id'` or `['user_id', 'date']` |
263258
| `duplicate_key` | Duplicate key columns | `'user_id'` or `['user_id', 'date']` |
264-
| `partitioned_by_expr` | Custom partition expression. Applies only if `partitioned_by` is not specified. | `'date_trunc(event_date, "month")'` |
265259
| `materialized_type` | Materialized type | `SYNC`, `ASYNC` |
266260
| `source_table` | Source table of synchronous materialized view | `schema_name`.`table_name` |
267261

@@ -333,12 +327,6 @@ FROM user_events
333327
GROUP BY user_id;
334328
```
335329

336-
### View Limitations
337-
338-
- `CREATE OR REPLACE VIEW` is not supported. If `replace=True`, SQLMesh will drop the view first and then create it
339-
- The `CASCADE` clause is not supported for dropping views
340-
- Materialized views support all Doris-specific properties and optimizations
341-
342330
## Schema Management
343331

344332
### Creating Schemas

sqlmesh/core/engine_adapter/doris.py

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,7 @@ def _get_data_objects(
122122
query = query.where(exp.func("LOWER", exp.column("table_name")).isin(*lowered_names))
123123

124124
result = []
125-
rows = self.fetchall(query)
126-
for schema_name, table_name, table_type in rows:
125+
for schema_name, table_name, table_type in self.fetchall(query):
127126
try:
128127
schema = str(schema_name) if schema_name is not None else str(schema_name)
129128
name = str(table_name) if table_name is not None else "unknown"
@@ -175,7 +174,7 @@ def create_view(
175174
view_name,
176175
query_or_df,
177176
target_columns_to_types,
178-
replace=False, # Already dropped if needed
177+
replace=False,
179178
materialized=False,
180179
materialized_properties=materialized_properties,
181180
table_description=table_description,
@@ -270,16 +269,16 @@ def _create_materialized_view(
270269
table_kind="MATERIALIZED_VIEW",
271270
)
272271

273-
create_exp = exp.Create(
274-
this=schema,
275-
kind="VIEW",
276-
replace=False,
277-
expression=query,
278-
properties=properties_exp,
272+
self.execute(
273+
exp.Create(
274+
this=schema,
275+
kind="VIEW",
276+
replace=False,
277+
expression=query,
278+
properties=properties_exp,
279+
)
279280
)
280281

281-
self.execute(create_exp)
282-
283282
def drop_view(
284283
self,
285284
view_name: TableName,

0 commit comments

Comments
 (0)