Feat/anthropic extended ttl

[md2k]

Description

Implements granular per-message-type caching for Anthropic models to improve token efficiency in Agent mode. Adds new CacheBehavior options to specify how many of each message type to cache (user messages, tool results, assistant tool calls, etc.) instead of only caching the last 2 user messages.
This is related to issue #6135

Checklist

• I've read the contributing guide
• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screenshots

N/A - Backend caching enhancement with no visual changes.

Tests

Added comprehensive test suite core/llm/llms/Anthropic.enhanced-caching.test.ts with 6 test cases covering:

• Tool result message caching
• Assistant tool call message caching
• Per-type caching limits validation
• Disabled caching behavior
• Fallback TTL handling
• Core shouldCacheMessage logic

All tests pass and validate the new per-type caching functionality while maintaining backward compatibility.

[netlify]

👷 Deploy request for continuedev pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	ad24f59

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

1 out of 2 committers have signed the CLA.
✅ (md2k)[https://github.com/md2k]
❌ @Yevgen Flerko (OSV)
Yevgen Flerko (OSV) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[recurseml]

😱 Found 3 issues. Time to roll up your sleeves! 😱

[md2k]

I have read the CLA Document and I hereby sign the CLA

[md2k]

Some details about how long session with big context looks with 5min cache and 1h cache and cost perspective:
5 min cache:

[md2k]

1h ttl

[sestinj]

This is a great PR as far as the code goes. I kind of want to step back though to better understand whether you think this could be a sensible default rather than a configuration option. I'm weary of too many options and if everyone would benefit from the way you are configuring your Anthropic models, maybe we should just ship that as the default (I repeated all this in a comment below)

[chezsmithy]

@sestinj maybe we align to this previous PR. #5371

It introduced a single caching setting that controls all the options. Whatever we do here I should likely bring to Bedrock as well.

Watching.

[sestinj]

Agreed @chezsmithy ! Thanks for linking the PR here, that what I had in mind

[md2k]

@chezsmithy thanks for link, i will take a look and will re-align my PR

[md2k]

Due some time constraint this week i have, will work thru PR closer to the end of week or over weekend.

[sestinj]

Sounds good!

[md2k]

@chezsmithy i looked into PR for AWS Bedrock, at this moment its support limited to 5min cache, what cachePoint: { type: "default" }, and you already have implemented system, user messages and tools. the only improvements i can see here, is to maybe extend internal logic to ensure we covering all types of messages similar to my Anthropic PR. So probably right now Bedrock not require any changes. also i need look into its api because some property names different from Anthropic.

[md2k]

@sestinj, i did more simple version with only following settings

        "cacheSystemMessage": true, -- original
        "cacheConversation": true,  -- original
        "cacheToolMessages": true, - new
        "useExtendedCacheTtlBeta": true,  -new
        "cacheTtl": "5m|1h" - new

the reason, while it still in beta feature i probably keep it as enabler, plus it costs 2x tokens, by this reason I allow to disable tools messages.
I going to work with this patch myself couple of days, to understand how it feels with some settings enabled/disabled and token/$$ consumption.

But if you have any suggestions, please, let me know.

[md2k]

found some issues in caching logic where we can hit maximum 4 slots allowed to be cached by Anthropic, so implementing some logic to ensure priority to what cache settings will be applied

[md2k]

while testing logic, i was able to achieve pretty nice results

520k and 620k input/cached tokens... unfortunately i screwed logic and burned around 400k input (no cache at all) and invalidated cache for 300k tokens. so real input/cache writes is around 200k/200k +- and 6.6mil cache read hits.

And costs (supposed to be cheaper, but due small mistake i burned for nothing close to 4$ due broken logic :D ):

[md2k]

new logic not committed yet. going to test it a bit more. Also added some debug to console. curious if possible to keep this debug or maybe make some option for stats (can be handy to see)

[md2k]

I found my problem 😀 I made it too complicated. I will try rolling back to my previous initial code and simplifying it. I spent a couple of hours reading through all of Anthropic's documentation and examples to understand how their breakpoints work and where and how to use them. Things were much easier than I thought. So, hopefully, I will have the final code in a few days. Then, we can benefit from a close to 90-95% cache hit rate.

[md2k]

@sestinj i tested current implementation and it looks good. so user can enable/disable tools/system/conversation cache breakpoints, and control 5 min/1 hour TTL when enabling Beta feature. Also added cacheDebug, it helps alot to understand what going on inside and how things cached and its efficiency.
If you do not want to keep debug, i can remove it.
And i asking for help because i can't resolve mdx conflict :(

[md2k]

i found also problem with CLA, looks like it will be more cleaner if i re-do entirely PR and apply my changes to latest main branch instead of trying to solve the puzzles with CLA and merge conflicts.

[md2k]

Seems for best efficient cost/cache ratio, Apply/Edit models better to disable cacheConversations, because apply/edit does not include history. while it make sense, i not sure why rules / system messages stripped out as well (i would prefer to include them to ensure that during edit/apply AI follow the same guidelines as during conversation), but the main point need to disable conersation caching for Edit/Apply to not waste $$$ and pay just for vanilla costs, with caching single input message we literally wasting x1.25 or x2.0 depends on cache type which written to cache but never used after since each edit/apply a new mini-session.

In any case i think i did the best from what i can to achieve best caching logic and support Anthropic extended ttl. It works perfect. Small issue with 1h ttl that it has strictly limited TTL and not using LRU style, so its invalidation will always happen after 1hour. It still good option anyway. I have idea maybe to implement sort of "ping" where run timer after last message and automatically send context +ping which will cost few tokens only but will refresh 5min ttl. The problem here it needs more testing and investigation about how long we can keep 5min TTL using ping approach and also have some fail-safe feature to not abuse anthropic functionality where user can forgot to close session and ping can stay for hours and days. So this is only idea.

Also code tested against current main branch with v1.1.56 of plugin. unfortunately i cant do CLA from another account, so maybe it easier to create new PR with rebased from current main and without leftish committers :D