← Graph

How to handle very large event volumes over time?

question 3 connections

Audience member asks how to deal with millions of events in a write-heavy system — keep them forever, snapshot, or compact? Pašalić: depends on business. Some shops use Kafka with log compaction (he doesn't recommend Kafka for this). Financial institutions can snapshot at year-end and archive to S3 or disk. Advertising domains can compact per-campaign once reports are delivered, since post-campaign click data becomes irrelevant. Time-travel then works only up to the compaction point. In e-commerce, archive events out of the transactional store rather than discarding them, so reports and benefits are preserved.

answer_summary
Strategy depends on business value: snapshot at meaningful points (year-end, campaign end), archive cold events to cheap storage, compact when data loses business value.
question How to handle very large event volumes over time?
about
Snapshotting concept
Answer revolves around snapshotting and compacting events at business-meaningful points.
question How to handle very large event volumes over time?
about
Answer mentions Kafka log compaction as one (not recommended) option.
question How to handle very large event volumes over time?
asked_at
Asked during the talk's Q&A.

Provenance

Read by
5 extractions