Tokio Runtime Worker Panick: Queue background task

[pandysp]

System Info

thread 'tokio-runtime-worker' panicked at core/src/queue.rs:87:14:
Queue background task dropped the receiver or the receiver is too behind. This is a bug.: "Full(..)"
stack backtrace:
   0:        0x1007dacb0 - __mh_execute_header
   1:        0x1004ceeec - __mh_execute_header
   2:        0x1007d9484 - __mh_execute_header
   3:        0x1007dab70 - __mh_execute_header
   4:        0x1007d91e8 - __mh_execute_header
   5:        0x10080a684 - __mh_execute_header
   6:        0x10080a5f4 - __mh_execute_header
   7:        0x10080ace4 - __mh_execute_header
   8:        0x100bdcc38 - __mh_execute_header
   9:        0x100bdced0 - __mh_execute_header
  10:        0x1008e81a0 - __mh_execute_header
  11:        0x1008e7568 - __mh_execute_header
  12:        0x1008e69c0 - __mh_execute_header
  13:        0x1008e601c - __mh_execute_header
  14:        0x10099c430 - __mh_execute_header
  15:        0x10099e21c - __mh_execute_header
  16:        0x10098e9f8 - __mh_execute_header
  17:        0x100992af4 - __mh_execute_header
  18:        0x10080bea4 - __mh_execute_header
  19:        0x18d454c08 - __pthread_cond_wait
[1]    7302 abort      RUST_BACKTRACE=full text-embeddings-router --model-id  --port 8001  4096  1

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

RUST_BACKTRACE=full text-embeddings-router \
  --model-id Qwen/Qwen3-Embedding-0.6B \
  --port 8001 \
  --max-batch-tokens 4096 \
  --max-concurrent-requests 1 \
  --auto-truncate

On MacBook Air M2 24GB 26.0.1 (25A362)

Expected behavior

No error

[pandysp]

I've had this same exact error message also on an Azure VM with docker image ghcr.io/huggingface/text-embeddings-inference:cpu-ipex-latest with

    command:
      [
        "--model-id",
        "Qwen/Qwen3-Embedding-0.6B",
        "--port",
        "8001",
        "--max-batch-tokens",
        "4096",
        "--auto-truncate",
        "--max-concurrent-requests",
        "2",
      ]

[alvarobartt]

Hey @pandysp thanks for reporting, this is being actively looked into, I'll let you know once solved so that you can run the latest image with the fix, hopefully soon, apologies for the inconvenience 🤗

[pandysp]

@alvarobartt thanks! Greatly appreciated 🙂👍🏻

[michaelfeil]

@pandysp most likley your backend died (deadlocked) here, creating backpressure. Probably a good idea to run a different backend.

[michaelfeil]

As in, your queue.rs had so much backpressure, and you never checked for /health. Idea would be to check for the health occationally e.g. in your k8s setup.

[michaelfeil]

Note, --max-concurrent-requests 1 is also really bad as it will enforce this error. #726 will help with this potentially. Its a race of double permit acuisiton.