Skip to content

fix: increase pruning interval #591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: increase pruning interval #591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ddcf623
add utxo store variables for interval and logging
Kartiiyer12 Sep 9, 2025
dda30cd
adding REMOVE_SPENT_UTXOS_BATCH_SIZE and increading the pruning interval
tienna Sep 9, 2025
3b4a895
REMOVE_SPENT_UTXOS enabled by default and block grace count set to 30…
tienna Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .env.docker-compose
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ SEARCH_LIMIT=100

## Yaci Indexer env
INDEXER_DOCKER_IMAGE_TAG=main
REMOVE_SPENT_UTXOS=false
#The number of safe blocks to keep in the store. 2160 blocks *(20 seconds/block in average)=4320 seconds=12 hours.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
REMOVE_SPENT_UTXOS=true
#The number of safe blocks to keep in the store. 129600 blocks *(20 seconds/block in average)=30 days.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=129600
REMOVE_SPENT_UTXOS_BATCH_SIZE=3000
BLOCK_TRANSACTION_API_TIMEOUT_SECS=5

YACI_SPRING_PROFILES=postgres,n2c-socket
Expand Down
7 changes: 4 additions & 3 deletions .env.docker-compose-preprod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ SEARCH_LIMIT=100

## Yaci Indexer env
INDEXER_DOCKER_IMAGE_TAG=main
REMOVE_SPENT_UTXOS=false
#The number of safe blocks to keep in the store. 2160 blocks *(20 seconds/block in average)=4320 seconds=12 hours.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
REMOVE_SPENT_UTXOS=true
#The number of safe blocks to keep in the store. 129600 blocks *(20 seconds/block in average)=30 days
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=129600
REMOVE_SPENT_UTXOS_BATCH_SIZE=3000
BLOCK_TRANSACTION_API_TIMEOUT_SECS=5

YACI_SPRING_PROFILES=postgres,n2c-socket
Expand Down
3 changes: 2 additions & 1 deletion .env.docker-compose-profile-advanced-level
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ DB_POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER=8
DB_POSTGRES_MAX_PARALLEL_WORKERS=16
DB_POSTGRES_SEQ_PAGE_COST=0.5
DB_POSTGRES_JIT=off
DB_POSTGRES_BGWRITER_LRU_MAXPAGES=100
DB_POSTGRES_BGWRITER_LRU_MAXPAGES=300
DB_POSTGRES_BGWRITER_DELAY=200ms
DB_POSTGRES_WAL_BUFFERS=512MB
DB_POSTGRES_CHECKPOINT_COMPLETION_TARGET=0.9
DB_POSTGRES_AUTOVACUUM_MAX_WORKERS=5
3 changes: 2 additions & 1 deletion .env.docker-compose-profile-mid-level
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ DB_POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER=4
DB_POSTGRES_MAX_PARALLEL_WORKERS=8
DB_POSTGRES_SEQ_PAGE_COST=1.0
DB_POSTGRES_JIT=off
DB_POSTGRES_BGWRITER_LRU_MAXPAGES=100
DB_POSTGRES_BGWRITER_LRU_MAXPAGES=200
DB_POSTGRES_BGWRITER_DELAY=200ms
DB_POSTGRES_AUTOVACUUM_MAX_WORKERS=5
6 changes: 4 additions & 2 deletions .env.h2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,10 @@ SEARCH_LIMIT=100

## Yaci Indexer env
INDEXER_DOCKER_IMAGE_TAG=main
REMOVE_SPENT_UTXOS=false
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
REMOVE_SPENT_UTXOS=true
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=129600
REMOVE_SPENT_UTXOS_BATCH_SIZE=3000

BLOCK_TRANSACTION_API_TIMEOUT_SECS=5

YACI_SPRING_PROFILES=h2,n2c-socket
Expand Down
8 changes: 5 additions & 3 deletions .env.h2-testdata
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,11 @@ SEARCH_LIMIT=100

## Yaci Indexer env
INDEXER_DOCKER_IMAGE_TAG=main
REMOVE_SPENT_UTXOS=false
#The number of safe blocks to keep in the store. 2160 blocks *(20 seconds/block in average)=4320 seconds=12 hours.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
REMOVE_SPENT_UTXOS=true
#The number of safe blocks to keep in the store. 129600 blocks *(20 seconds/block in average)=30 days.
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=129600
REMOVE_SPENT_UTXOS_BATCH_SIZE=3000

BLOCK_TRANSACTION_API_TIMEOUT_SECS=5

YACI_SPRING_PROFILES=h2-testdata,n2c-socket
Expand Down
4 changes: 2 additions & 2 deletions ...src/main/java/org/cardanofoundation/rosetta/api/block/service/LedgerBlockServiceImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,10 @@ public class LedgerBlockServiceImpl implements LedgerBlockService {
@Value("${cardano.rosetta.BLOCK_TRANSACTION_API_TIMEOUT_SECS:5}")
private int blockTransactionApiTimeoutSecs;

@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS:false}")
@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS:true}")
private boolean isRemovalOfSpentUTxOsEnabled;

@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:2160}")
@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:129600}")
private int removeSpentUTxOsLastBlocksGraceCount;

@PostConstruct
Expand Down
2 changes: 1 addition & 1 deletion api/src/main/java/org/cardanofoundation/rosetta/api/network/service/NetworkServiceImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ public class NetworkServiceImpl implements NetworkService {
@Value("${cardano.rosetta.SYNC_GRACE_SLOTS_COUNT:100}")
private int allowedSlotsDelta;

@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS:false}")
@Value("${cardano.rosetta.REMOVE_SPENT_UTXOS:true}")
private boolean isRemovalOfSpentUTxOsEnabled;

@PostConstruct
Expand Down
2 changes: 1 addition & 1 deletion api/src/main/resources/config/application-offline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ cardano:
rosetta:
OFFLINE_MODE: true
SYNC_GRACE_SLOTS_COUNT: ${SYNC_GRACE_SLOTS_COUNT:100}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:false}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:true}

YACI_HTTP_BASE_URL: ${YACI_HTTP_BASE_URL:http://localhost:9095}
HTTP_CONNECT_TIMEOUT_SECONDS: ${HTTP_CONNECT_TIMEOUT_SECONDS:5}
Expand Down
5 changes: 3 additions & 2 deletions api/src/main/resources/config/application.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,9 @@ cardano:
OFFLINE_MODE: ${OFFLINE_MODE:false}

SYNC_GRACE_SLOTS_COUNT: ${SYNC_GRACE_SLOTS_COUNT:100}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:false}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:2160}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:true}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:129600}
REMOVE_SPENT_UTXOS_BATCH_SIZE: ${REMOVE_SPENT_UTXOS_BATCH_SIZE:3000}
BLOCK_TRANSACTION_API_TIMEOUT_SECS: ${BLOCK_TRANSACTION_API_TIMEOUT_SECS:5}

YACI_HTTP_BASE_URL: ${YACI_HTTP_BASE_URL:http://localhost:9095}
Expand Down
5 changes: 3 additions & 2 deletions api/src/test/resources/config/application-test-integration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@ cardano:
SEARCH_LIMIT: ${SEARCH_LIMIT:100}
OFFLINE_MODE: ${OFFLINE_MODE:false}
SYNC_GRACE_SLOTS_COUNT: ${SYNC_GRACE_SLOTS_COUNT:100}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:false}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:2160}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS:true}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:129600}
REMOVE_SPENT_UTXOS_BATCH_SIZE: ${REMOVE_SPENT_UTXOS_BATCH_SIZE:3000}
BLOCK_TRANSACTION_API_TIMEOUT_SECS: ${BLOCK_TRANSACTION_API_TIMEOUT_SECS:5}

YACI_HTTP_BASE_URL: http://localhost:9095/api/v1
Expand Down
2 changes: 2 additions & 0 deletions docker-compose-indexer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@ services:
GENESIS_CONWAY_PATH: ${GENESIS_CONWAY_PATH}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT}
REMOVE_SPENT_UTXOS_BATCH_SIZE: ${REMOVE_SPENT_UTXOS_BATCH_SIZE}
BLOCK_TRANSACTION_API_TIMEOUT_SECS: ${BLOCK_TRANSACTION_API_TIMEOUT_SECS}
CARDANO_NODE_SOCKET_PATH: ${CARDANO_NODE_SOCKET_PATH}
HOST_N2C_SOCAT_HOST: ${HOST_N2C_SOCAT_HOST}
HOST_N2C_SOCAT_PORT: ${HOST_N2C_SOCAT_PORT}
SEARCH_LIMIT: ${LIMIT}
CONTINUE_PARSING_ON_ERROR: ${CONTINUE_PARSING_ON_ERROR}
PEER_DISCOVERY: ${PEER_DISCOVERY}
LOG: ${LOG}
volumes:
- ${CARDANO_CONFIG}:/config
- ${CARDANO_NODE_DIR}:${CARDANO_NODE_DIR}
Expand Down
2 changes: 2 additions & 0 deletions docker-compose-integration-test-indexer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@ services:
GENESIS_CONWAY_PATH: ${GENESIS_CONWAY_PATH}
REMOVE_SPENT_UTXOS: ${REMOVE_SPENT_UTXOS}
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT: ${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT}
REMOVE_SPENT_UTXOS_BATCH_SIZE: ${REMOVE_SPENT_UTXOS_BATCH_SIZE}
BLOCK_TRANSACTION_API_TIMEOUT_SECS: ${BLOCK_TRANSACTION_API_TIMEOUT_SECS}
CARDANO_NODE_SOCKET_PATH: ${CARDANO_NODE_SOCKET_PATH}
HOST_N2C_SOCAT_HOST: ${HOST_N2C_SOCAT_HOST}
HOST_N2C_SOCAT_PORT: ${HOST_N2C_SOCAT_PORT}
SEARCH_LIMIT: ${LIMIT}
CONTINUE_PARSING_ON_ERROR: ${CONTINUE_PARSING_ON_ERROR}
PEER_DISCOVERY: ${PEER_DISCOVERY}
LOG: ${LOG}
volumes:
- ${CARDANO_CONFIG}:/config
- ${CARDANO_NODE_DIR}:${CARDANO_NODE_DIR}
Expand Down
5 changes: 3 additions & 2 deletions docker/.env.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,10 @@ PRINT_EXCEPTION=true

## Yaci Indexer env
YACI_SPRING_PROFILES=postgres,n2c-socket
REMOVE_SPENT_UTXOS=false
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=2160
REMOVE_SPENT_UTXOS=true
REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT=129600
BLOCK_TRANSACTION_API_TIMEOUT_SECS=5
REMOVE_SPENT_UTXOS_BATCH_SIZE=3000

# database profiles: h2, h2-testdata, postgres
MEMPOOL_ENABLED=false
Expand Down
11 changes: 8 additions & 3 deletions yaci-indexer/src/main/resources/application.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,14 @@ logging.level.com.bloxbean.cardano.yaci.core.protocol.keepalive=debug
store.admin.auto-recovery-enabled=true
store.admin.health-check-interval=20

store.utxo.pruning-enabled=${REMOVE_SPENT_UTXOS:false}
store.utxo.pruning-safe-blocks=${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:2160}
store.utxo.pruning-interval=600
store.utxo.pruning-enabled=${REMOVE_SPENT_UTXOS:true}
store.utxo.pruning-safe-blocks=${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:129600}
store.utxo.pruning-interval=10800
store.utxo.pruning-batch-size=${REMOVE_SPENT_UTXOS_BATCH_SIZE:3000}
logging.level.com.bloxbean.cardano.yaci.store=${LOG:error}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should not be necessary actually, confusing. I wouldn't do this, it should take root logger

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LOG env variable is the already existing. The logging was just added as we were not seeing the Yaci-indexer pruning logs and it needed to be turned to debug to see the logs. Could you clarify, is LOG env variable not root logging option?

spring.datasource.hikari.maximum-pool-size=20
spring.datasource.hikari.minimum-idle=5


store.epoch.endpoints.epoch.local.enabled=true

Expand Down
5 changes: 3 additions & 2 deletions yaci-indexer/src/test/resources/application-test-integration.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,6 @@ spring.h2.console.enabled=true
store.cardano.host=${CARDANO_NODE_HOST:localhost}
store.cardano.port=${CARDANO_NODE_PORT:8090}

store.utxo.pruning-interval=600
store.utxo.pruning-safe-blocks=${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:2160}
store.utxo.pruning-interval=3600
store.utxo.pruning-safe-blocks=${REMOVE_SPENT_UTXOS_LAST_BLOCKS_GRACE_COUNT:129600}
store.utxo.pruning-batch-size=${REMOVE_SPENT_UTXOS_BATCH_SIZE:3000}
Loading