Discussions

• 1 You must be logged in to vote

  🙏

  Determining Overall Speed for One Long Prompt

  chigkim announced Apr 30, 2025 in Q&A

  

  

  

  

  7
• 5 You must be logged in to vote

  💬

  Structured Generation with Reasoning Parser in offline mode.

  psych0v0yager announced May 4, 2025 in General

  

  

  

  

  5
• 1 You must be logged in to vote

  💬

  vLLM cannot connect to existing Ray cluster

  as-bain announced May 1, 2025 in General

  

  

  

  

  3
• 2 You must be logged in to vote

  🙏

  Any known integration with n8n ?

  wugutech announced Apr 28, 2025 in Q&A

  

  

  

  

  2
• 2 You must be logged in to vote

  🙏

  Why temperature=0,top_p=1,seed=42 is still not enough to fix the llm's output!?

  beyondguo announced Apr 25, 2025 in Q&A

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  ......lib/python3.12/site-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

  Rex818 announced May 1, 2025 in Q&A

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  How to increase context lenght and make things work

  kresimirfijacko announced Apr 3, 2024 in Q&A

  

  

  1
• 10 You must be logged in to vote

  🙌

  How to benchmark vLLM a short tutorial

  samos123 announced Aug 6, 2024 in Show and tell

  

  

  

  

  5
• 2 You must be logged in to vote

  🙏

  How does the profile_run work?

  zhuangqh announced Nov 7, 2024 in Q&A

  

  

  

  

  9
• 6 You must be logged in to vote

  🙏

  How to: custom attention mask? Specifically, bidirectional attention for context.

  ojus1 announced Jun 3, 2024 in Q&A

  

  

  1
• 3 You must be logged in to vote

  🙏

  How to only download model without serving it?

  yardenshoham announced Dec 26, 2024 in Q&A

  

  

  

  

  2
• 9 You must be logged in to vote

  🙏

  Why do vllm set default keep-alive timeout to 5s?

  SeungminHeo announced Sep 9, 2024 in Q&A

  

  

  1
• 4 You must be logged in to vote

  🙏

  KV cache in VLLM serve

  farouk09 announced Apr 23, 2025 in Q&A

  

  

  1
• 3 You must be logged in to vote

  🙏

  RAG LLM Generating the entire Prompt along with Question in the generated response

  Arpx17 announced Feb 29, 2024 in Q&A

  

  

  

  

  4
• 2 You must be logged in to vote

  🙏

  Why is the GPU KV cache usage very low?

  usage How to use vllm

  tammypi announced Jun 18, 2024 in Q&A

  

  

  

  

  4
• 9 You must be logged in to vote

  🙏

  Measuring Token Cache Hits in vLLM

  LeonardTwcs announced Mar 10, 2025 in Q&A

  

  

  

  

  3
• 8 You must be logged in to vote

  📣

  GitHub discussion is not used anymore, please use the forum for discussion.

  youkaichao announced Mar 20, 2025 in Announcements

  

  

  

  

  15
• 0 You must be logged in to vote

  🙏

  Help serving vllm with kuberay

  torsteinelv announced Jan 11, 2025 in Q&A

  

  

  

  

  8
• 13 You must be logged in to vote

  🙏

  vLLM generating repeated/duplicate responses

  saraswatmks announced Jan 21, 2025 in Q&A

  

  

  

  

  8
• 4 You must be logged in to vote

  🙏

  Generation speed suddenly became very slow, what might be the cause?

  zhuwenfei-wintech announced Sep 27, 2024 in Q&A

  

  

  

  

  3
• 4 You must be logged in to vote

  🙏

  LLaVA with GGUF files (and --mmproj argument)?

  benmayersohn announced Sep 10, 2024 in Q&A

  

  

  1
• 4 You must be logged in to vote

  🙏

  MultiModal model batched inference

  aswad546 announced Feb 24, 2025 in Q&A

  

  

  3
• 17 You must be logged in to vote

  🙏

  Is there a way to set max-concurrency

  tommasofavaron1 announced Jan 31, 2024 in Q&A

  

  

  

  

  2
• 3 You must be logged in to vote

  🙏

  How can I log user id from the API request

  vinayK34 announced Feb 6, 2025 in Q&A

  

  

  1
• 1 You must be logged in to vote

  🙏

  Can vllm serving clients by using multiple model instances?

  aoyulong announced Jun 21, 2023 in Q&A

  

  

  

  

  7