← Graph

Sidekiq Batches Lightning Talk

talk 5 connections

wroclove.rb 2022 lightning talk prompted by a Q&A question during Paweł Dąbrowski's Sidekiq talk. The speaker describes using Sidekiq batches once in her four years of Ruby work to execute commands against each of ~1000 rows in a source file. Approach: parse the file and enqueue one worker per row inside a Sidekiq batch, passing a unique identifier (time + name + sequence number). Each worker writes its result to Redis under a key derived from the source name and sequence. A batch callback runs one final 'finish result' worker that iterates from 1 to the known row count, reads each Redis key, and produces a single result file with the same sequence order — 'and business is happy'.

type
lightning-talk
talk Sidekiq Batches Lightning Talk
about
Sidekiq tool
Walks through parallelizing per-row file processing using Sidekiq batches.
talk Sidekiq Batches Lightning Talk
about
Sidekiq Batches concept
Core subject of the talk.
talk Sidekiq Batches Lightning Talk
about
Redis tool
Each worker writes its per-row result to Redis under a sequence key; a final callback worker reads from Redis to assemble the output file.
talk Sidekiq Batches Lightning Talk
presented_at
Lightning talk at the wroclove.rb 2022 conference.
talk Sidekiq Batches Lightning Talk
related_to
Speaker says she remembered this approach because of a Sidekiq batches question raised during Paweł Dąbrowski's Sidekiq talk.

Provenance

Created in
wroclove.rb 2022 Lightning Talks 2026-04-17 18:12
Read by
18 extractions