Autocomplete requests get cancelled on local server

[seanxwzhang]

Hey great job with continue, been playing with it for the past day but couldn't get autocomplete working for my custom openai-compatible server.

when I start the server using vllm serve bigcode/starcoder2-3b, and using the following config at my ~/.continue/config.yaml

name: Local Agent
version: 1.0.0
schema: v1
models:
  - name: Mycoder
    provider: vllm
    model: bigcode/starcoder2-3b
    apiBase: http://mybox:8000/v1
    roles:
      - autocomplete

I'm getting valid requests on my servers, as shown below

cache usage: 0.0%, Prefix cache hit rate: 74.0%
(APIServer pid=3260928) INFO:     100.87.22.106:60817 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60820 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60824 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60826 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60827 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60828 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60829 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO 09-17 01:31:43 [loggers.py:123] Engine 000: Avg prompt throughput: 22.6 tokens/s, Avg generation throughput: 16.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 65.6%
(APIServer pid=3260928) INFO:     100.87.22.106:60831 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60832 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60833 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60837 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO:     100.87.22.106:60840 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=3260928) INFO 09-17 01:31:53 [loggers.py:123] Engine 000: Avg prompt throughput: 18.6 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 61.9%
(APIServer pid=3260928) INFO 09-17 01:32:03 [loggers.py:123] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 61.9%

but the autocomplete on the vscode side doesn't quite work despite the requests finished. Upon looking at the continue console, I saw most of the requests were cancelled

Why are these requests getting cancelled?