Skip to content

bug: Transactional topic example exhausts producer_id limit with random IDs #700

@krasnovdm

Description

@krasnovdm

Bug Report

YDB Python SDK version:

Environment

Environment is irrelevant for reproduction - issue is in code example

Current behavior:

In the transactional topic write example (topic_transactions_async_example.py):

  1. Each call to tx_writer without explicit producer_id generates a new random producer_id
  2. Transaction retries create additional producer_ids
  3. This rapidly consumes the producer_id limit (100K per 14 days)
  4. The example encourages production-unsafe usage patterns

Expected behavior:

Examples should demonstrate best practices:

  1. Explicit producer_id should be used to avoid random ID generation
  2. Example should be resilient to transaction retries
  3. Should show either fixed producer_id or partition_id usage
  4. Should not risk exhausting producer_id limits in production scenarios

Steps to reproduce:

  1. Use example from topic_transactions_async_example.py
  2. Run write_with_tx_example method in a loop or with frequent retries
  3. Observe rapid growth of producer_ids in the topic

Related code:
https://github.com/ydb-platform/ydb-python-sdk/blob/main/examples/topic/topic_transactions_async_example.py#L28

async def write_with_tx_example(driver: ydb.aio.Driver, topic: str, message_count: int = 10):
    async with ydb.aio.QuerySessionPool(driver) as session_pool:

        async def callee(tx: ydb.aio.QueryTxContext):
            tx_writer: ydb.TopicTxWriterAsyncIO = driver.topic_client.tx_writer(tx, topic)

            for i in range(message_count):
                async with await tx.execute(query=f"select {i} as res;") as result_stream:
                    async for result_set in result_stream:
                        message = str(result_set.rows[0]["res"])
                        await tx_writer.write(ydb.TopicWriterMessage(message))
                        print(f"Message {result_set.rows[0]['res']} was written with tx.")

        await session_pool.retry_tx_async(callee)

Other information:

Current example can exhaust 100K producer_id limit under moderate load

Possible Solutions
Fixed producer_id
Explicit partition selection

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions