v2.5.0

What's Changed

Features

Checkpointing

Checkpointing is an overhaul of the previous commit structure. It is meant to better synchronize processing progress (i.e. committing topic offsets) and state updates to ensure consistency of the state.

It should also increase processing speed anywhere from 1.3x-2.5x due to its new batched commit approach.

To adjust this new commit frequency, users can set a (new) commit_interval (Default: 5 seconds):

app = Application(commit_interval=5)

For more details, see the Checkpoint docs.

GroupBy

GroupBy enables users to "group" or "re-key" their messages based on the message value, typically to perform (stateful) aggregations on them (much like SQL).

With the new StreamingDataFrame.group_by(), you can do this while including other StreamingDataFrame operations before or after (so only one Application is needed):

# data: {"user_id": "abc", "int_field": 5}
app = Application()
sdf = app.dataframe()
sdf["new_col"] = sdf["int_field"] + 1
sdf = sdf.group_by("user_id")
sdf = sdf.apply(lambda r: r["new_col"])
sdf = sdf.tumbling_window(duration_ms=3600).sum().final()
# ...etc...

Users can group by a column name, or provide a custom grouping function.

For more details, see the GroupBy docs.

Enhancements

• Docs updates by @stereosky in #344, #352
• add default error cb to Admin by @tim-quix in #343

Full Changelog: v2.4.2...v2.5.0

v2.4.2

What's Changed

• Fix handling of topics created outside of Quix Cloud by @tim-quix in #338

• Add clearer error messages for invalid SDF column name references by @tim-quix in #322

• Better handling of topic creation errors in Quix Cloud by @tim-quix in #337

• Use pyproject.toml instead of setup.cfg by @tim-quix in #339

• Update docs by @tbedford #325 #331

• Update README by @stereosky #332 #335

Full Changelog: v2.4.1...v2.4.2

v2.4.1

What's Changed

• Fix 404s in README by @daniil-quix in #328
• hotfix bug around undefined workspace id for Quix API class by @tim-quix in #329

Full Changelog: v2.4.0...v2.4.1

v2.4.0

What's Changed

Features

Unified Application and Application.Quix() by @tim-quix in #313

In previous versions, to connect to Quix Cloud brokers you needed to use a separate factory method to create an Application - Application.Quix().
Due to that, developing apps with local Kafka and deploying them to Quix Cloud required additional code changes.
In this release, we improved that, and you now may use a single Application() class with different settings to connect both to Quix Cloud and standalone Kafka brokers.

See more about working with Quix Cloud in the docs - Connecting to Quix Cloud

Enhancements

• Improved Quix API error handling by @tim-quix in #320
• Adjust producer flush docstring by @tim-quix in #321
• New docs by @daniil-quix in #315
• Update black formatter version by @daniil-quix in #316

Full Changelog: v2.3.3...v2.4.0

v2.3.3

What's Changed

• Example Bugfix: Do not override state input on each update by @alexmorley in #310
• Fix default replication_factor for topics created by Quix apps by @daniil-quix in #317

New Contributors

• @alexmorley made their first contribution in #310

Full Changelog: v2.3.2...v2.3.3

v2.3.2

What's Changed

Features

• Support GZIP compression of messages formatted in Quix formats by @peter-quix in #305

Fixes

• Fix type hints and code completion by @tim-quix in #292
• Allow processing of non-dict values in StreamingDataFrame by @harisbotic in #307
• Skip processing of messages with None keys in windowed aggregations by @harisbotic in #306

Enhancements

• Set the default value of consumer group to 'quixstreams-default' in Application by @harisbotic in #299
• Set a default value for Quix__Portal__Api env variable in Quix apps by @harisbotic in #294
• Raise exception on __bool__ checks in StreamingDataFrame by @tim-quix in #295
• Don't set ssl.endpoint.identification.algorithm anymore for Quix apps by @harisbotic in #293
• Update docs by @stereosky in #300

Full Changelog: v2.3.1...v2.3.2

v2.3.1

What's Changed

• Added support for Changelog topics

  • Changelog topics provide fault tolerance capabilities to state stores.
    Each state store now has a corresponding changelog topic to keep track of the state updates in Kafka.
  • Changelog topics are enabled by default and can be disabled.
  • See more about changelog topics in the docs

• Application.run() class now verifies that topics exist before starting the application.
  If topics don't exist, the Application instance will try to create them automatically if auto_create_topics is set to True (default).
  The topic parameters can also be specified, see more in the docs

• This is the first non-alpha release of Quix Streams v2. It can now be installed from pip without the --pre flag.

Breaking changes

• The partition assignment strategy is now always set to cooperative-sticky and cannot be configured anymore because the consumer relies on the incremental_assign() API for recovery.
  Previously, the assignment strategy was set to range by default, and range is a non-cooperative strategy.
  Since cooperative and non-cooperative (eager) strategies must not be mixed, all consumers in the group must first leave the group, and then join it again after upgrading the application to this version.

Full Changelog: v2.2.1a...v2.3.1

v2.2.1a

What's Changed

Bugfixes

• Updates to work with Redpanda brokers via Quix platform by @daniil-quix in #283
• Fix RocksDB does not retry opening if it's locked by another process by @harisbotic in #272
• Add name property to window s by @harisbotic in #277

Docs

• Add docs about windows by @daniil-quix in #274
• Update docs config by @daniil-quix in #276
• Fix windows docs by @daniil-quix in #278
• Update windows docs by @stereosky in #279
• Update readme by @daniil-quix in #284
• add a line to the readme about sliding windows by @tim-quix in #285

New Contributors

• @stereosky made their first contribution in #279

Full Changelog: v2.2.0a...v2.2.1a

v2.2.0a

What's Changed

Added Windowed Aggregations

• Implemented two types of time-based windows:

  • Tumbling windows slice time into even non-overlapping periods.
    Example: [0, 10), [10, 20), [20,30)

  • Hopping windows slice time into even overlapping periods with a fixed step.
    Example: [0,10), [1, 11), [2, 12)

• Support for various aggregation functions: sum, count, mean, min, max, reduce
• Two modes of emitting aggregated results:
  • final - to emit results when the window is closed
  • current - to emit results for each incoming message as it's processed
• Support for out-of-order processing

Find more about Windowed Aggregations in the docs

Full Changelog: v2.1.4a...v2.2.0a

v2.1.4a

What's Changed

• Add support for Column families by @daniil-quix in #268
• Update the mapping of broker settings for Quix apps by @daniil-quix in #270
• Fix bug in RocksDB transaction on deleting cached key by @harisbotic in #269
• Add API to create a pre-configured Producer and Consumer on the Application level by @harisbotic in #261

Full Changelog: v2.1.3a...v2.1.4a