Autocomplete in VSCode not working

[StreamOfRon]

Continue: 1.2.2
Model: qwen2.5-coder:3b

I've been trying for the last couple of weeks to get autocomplete working in VSCode to no avail. When I get a completion out of it, it's more like the model's been asked to describe what it's looking at than an autocomplete:

I've been over the docs numerous times but whatever I'm missing hasn't been apparent yet.

I am running it via OpenWebUI -> Llama.cpp but when I've tested having it go to llama.cpp directly the behavior is identical so that seems to rule out that as a cause. Chat features work great using the gpt-oss:20b model this way as well. Config is as follows:

name: Local Assistant
version: 1.0.0
schema: v1
models:
  - name: Local
    provider: openai
    apiBase: https://<my-openwebui-url>/api
    apiKey: ${{ secrets.OPENWEBUI_KEY }}
    model: AUTODETECT
    useLegacyCompletionsEndpoint: false
    defaultCompletionOptions:
      contextLength: 16384
    roles:
      - chat
      - edit
      - apply
      - autocomplete
    requestOptions:
      timeout: 3000
    autocompleteOptions:
      disable: false
      maxPromptTokens: 1024
      debounceDelay: 250
      modelTimeout: 500
      maxSuffixPercentage: 0.2
      prefixPercentage: 0.3
      onlyMyCode: false
      useCache: true
      useImports: true
      useRecentlyEdited: true
      useRecentlyOpened: true

context:
  - provider: file
  - provider: code
  - provider: diff
  - provider: currentFile
  - provider: terminal
  - provider: open
  - provider: tree
  - provider: problems

[Aroni525]

Have the same issue. I didn't use autocomplete for a while, enabled it today to get surprised that the autocomplete suggestions became more like conversation snippets instead of context-aware code completions...

I get those all the time, no actual code completions. Tried to experiment with different models, providers... all resulting the same, just those conversational snippets.

[sestinj]

@Aroni525 would you be able to share your config.yaml. This would help me give better guidance or quickly come up with a fix if needed

[StreamOfRon]

@Aroni525 The autocompletions are a little better if I use their in-house "instinct" model, but ONLY in NextEdit mode, and only if I turn the temperature up (add temperature: 0.6 under defaultCompletionOptions). If I look at the Continue console it looks like it just isn't sending a prompt at all to other models -- just the context of the edit.

I'll leave it with Instinct for the time being because it at least kinda works but hopefully the bug in continue can get fixed -- I'd prefer to use qwen2.5-coder if only for speed

[sestinj]

@StreamOfRon a) Glad to hear that Instinct worked well for you! b) Very much would like to help you get to your preferred setup with qwen.

I tested out a few scenarios and it looks like the issue is that OpenWebUI doesn't support a raw completions endpoint, only /chat/completions (looks like you've caught this with useLegacyCompletionsEndpoint: false). This means that any request sent through their API will be formatted with chat tokens. Without full control over the prompt we have no ability to prompt for the FIM use case. What's probably happening is that we are actually nesting the FIM special tokens inside of the chat special tokens. I've actually asked them about this at some point and I don't think it is in the plans, though I'm not sure whether there might be a way to disable all formatting on their side.

The easiest alternative would be Ollama, where you only need to add uses: ollama/qwen2.5-coder-3b to your config.yaml. I've tested this to be certain it works. But if you have another provider that can also support a raw /v1/completions endpoint then we can work with that.

[StreamOfRon]

The problem with ollama is that they neither support the embedded AMD GPU on my host (Ryzen AI 9 HX370 -- it doesn't get picked up even with the environment variable workarounds) and they've refused to support Vulkan, which is giving me respectable performance.

I'm also wanting to expose the endpoint to the outside so I can access it when I'm not at home and need something that'll provide token auth, which is why I'm trying to pipe everything through Open-WebUI. If you have another suggestion for a way to provide openai-api-compatible auth as a proxy in front of llama.cpp/llama-swap (since neither has any apparent support to handle auth), I'm open to that, of course.

I'm curious, when you spoke to them, is the raw endpoint something they were opposed to supporting, or just didn't want to prioritize?

[StreamOfRon]

It does look like open-webui allows me to override the system prompt for the model, and since I don't intend to use a lightweight coder model for chat I'm definitely interested in trying that

[StreamOfRon]

@sestinj ok definitely getting closer -- I set up a reverse proxy that'll do this in a quick-and-dirty way and I am now getting reasonable autocompletions from qwen2.5-coder. Main problem now is multi-line. In the console I am seeing that qwen sent back:

securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

But I was only offered to add securityContext: and the result on the console is "Cancelled"