← Graph

Use async for Heavy IO LLM Workloads

takeaway 2 connections

For LLM-heavy, IO-bound background work in Ruby, prefer fibers over threads. The async community provides an adapter for Rails Active Job: change the adapter to async and jobs benefit from fiber-based concurrency. async's barrier abstraction schedules concurrent IO tasks and by default holds back new work once the system hits ~80% load, cancelling unbounded tasks; you can also manage concurrency manually via a semaphore with a fixed number of slots, which is handy inside a microservice world to throttle outbound calls and avoid local DoS on upstream services. In the speaker's production case, switching from sequential calls to async cut a workflow from above 50 seconds down to roughly 15–20 seconds.

type
recommendation
takeaway Use async for Heavy IO LLM Workloads
about
async tool
Recommends adopting the async gem for LLM workflows.
takeaway Use async for Heavy IO LLM Workloads
from_talk
Adam Okoń's recommendation to adopt async for LLM-heavy IO workloads.

Provenance