← Graph

From PostgreSQL to SQLite in Rails

talk 31 connections

Wojtek Wrona's wroclove.rb 2025 single-speaker talk describing a real migration of a production Rails application (18,000 lines of code, twice as many test lines; web + mobile front ends; a few hundred daily users) from PostgreSQL on Render to SQLite deployed on a single Hetzner server via Hatchbox. Motivations: small team (8 devs, 3 backend), tight budgets, no dedicated DevOps, a desire for simpler infrastructure and lower cost. Inspired by Stephen Margheim's wroclove.rb 2024 talk, enabled by improved SQLite support in Rails and the Solid Stack (Solid Cache, Solid Queue). Benefits reported: request times roughly halved, infrastructure cost dropped by an order of magnitude (~$77/mo vs ~$650/mo for equivalent Render capacity), end-to-end mobile tests can run in parallel by copying the database file, no more Docker in the pipeline, bundle install is the full setup. Migration strategy: keep using Postgres while making the schema SQLite-compatible in small reversible steps — (1) replace unsupported column types (inet, uuid, interval, arrays) with custom Active Record types that serialize/deserialize Ruby objects (arrays stored as JSON so they remain queryable); (2) adopt the Solid Stack (each solid gem uses its own SQLite file, runnable alongside Postgres); (3) drop problematic check constraints that use Postgres-specific function semantics (casting etc.) and re-add them after the switch (sqlpkg/similar tools translate DDL); then (4) do the actual data migration via a tweaked open-source bash script, point database.yml to a new SQLite file, run the test suite, fix breakages. Trade-offs encountered: ILIKE is unavailable and SQLite's default LIKE is ASCII-only case-insensitive, fixed via the sqlean text extension plus column COLLATE NOCASE or stored virtual columns for combined lowercase-indexed fields (GLOB as the case-sensitive counterpart); JSON array querying required switching to SQLite JSON functions (abstracted per Margheim's blog post); NULL ordering differs between Postgres and SQLite (must specify NULLS FIRST/LAST explicitly); concurrency — WAL mode allows many readers but only one writer, and immediate-transaction mode means long transactions or transactions wrapping external-service calls lock the database, which is especially painful for data migrations (Rails wraps each migration in a transaction; mitigations are raw SQL, disabling the migration transaction, or moving data migration into a rake task); foreign-key errors are opaque (only an error code) — debug by disabling FKs, running the query, and executing PRAGMA foreign_key_check in development; single-node scaling requires going vertical (Hetzner offers up to 48 dedicated vCPUs with 192 GB RAM; other clouds up to 256 vCPUs; bare metal gives full NVMe SSD access) or using libSQL (Turso's fork, with Ruby adapter 'ruby-libsql' and embedded replicas that forward writes to the primary, now also supporting sync to AWS S3 Express). SQLite extensions are distributed via the 'sqlpkg' gem (bundles common ones like IP, math, regex; loadable via database.yml in Rails main); custom scalar and aggregate Ruby functions can be defined and called directly from SQL since SQLite runs in the Ruby process. Q&A: backups via Litestream streaming every second to Cloudflare R2 (near-free, point-in-time recovery within last 48 hours, one-command replication to dev) plus a daily cron running SQLite's built-in backup command exported to S3; using Solid Queue with batches for heavy transactional data migrations is viable; why not run a local Postgres — still has serialization overhead, connection pool management, and more DevOps, whereas SQLite is meant to be embedded; migration took ~3 days of one person's work ~3 months before the talk, downtime ~1–2 hours, no regrets; largest table tens of thousands of rows, total DB just over 1 GB; N+1 is less costly on the query side but Active Record's per-row allocation overhead remains so N+1 elimination is still needed.

type
talk
talk From PostgreSQL to SQLite in Rails
about
SQLite tool
Primary subject: migrating a Rails app to SQLite.
talk From PostgreSQL to SQLite in Rails
about
PostgreSQL tool
Migration source database; many trade-offs are discussed relative to Postgres.
talk From PostgreSQL to SQLite in Rails
about
Migration is done inside a Rails application using Active Record.
talk From PostgreSQL to SQLite in Rails
about
Solid Cache project
Adopted as part of the Solid Stack during the migration.
talk From PostgreSQL to SQLite in Rails
about
Solid Queue project
Adopted as part of the Solid Stack during the migration.
talk From PostgreSQL to SQLite in Rails
about
Litestream tool
Used for continuous backup/replication to Cloudflare R2.
talk From PostgreSQL to SQLite in Rails
about
Hatchbox tool
Used to deploy the Rails app to a single Hetzner node.
talk From PostgreSQL to SQLite in Rails
about
Render tool
Previous hosting platform replaced after migration.
talk From PostgreSQL to SQLite in Rails
about
Docker tool
Docker removed from the pipeline after the SQLite migration.
talk From PostgreSQL to SQLite in Rails
about
libSQL tool
Presented as the path to multi-node SQLite when vertical scaling isn't enough.
talk From PostgreSQL to SQLite in Rails
about
sqlpkg tool
Extension packaging gem discussed for bundling SQLite extensions.
talk From PostgreSQL to SQLite in Rails
about
Used to serialize Postgres-specific types for SQLite.
talk From PostgreSQL to SQLite in Rails
about
Core methodology of the talk — schema first, data second, all reversible.
talk From PostgreSQL to SQLite in Rails
about
Case-insensitive comparisons via collations and stored virtual columns.
talk From PostgreSQL to SQLite in Rails
about
Demoed scalar and aggregate functions written in Ruby.
talk From PostgreSQL to SQLite in Rails
about
Discussed in depth with migration-specific mitigations.
talk From PostgreSQL to SQLite in Rails
about
Largest frustration of the migration; PRAGMA foreign_key_check workaround.
talk From PostgreSQL to SQLite in Rails
about
Explains how SQLite scales on modern hardware.
talk From PostgreSQL to SQLite in Rails
about
WAL mode is the Rails default and underlies SQLite's concurrent reads.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
asked_at
From PostgreSQL to SQLite in Rails talk
Audience Q&A.
person Wojtek Wrona
authored
From PostgreSQL to SQLite in Rails talk
Wrona presented this single-speaker talk at wroclove.rb 2025.
from_talk
From PostgreSQL to SQLite in Rails talk
Top-level recommendation of the talk.
from_talk
From PostgreSQL to SQLite in Rails talk
Key methodological takeaway.
talk From PostgreSQL to SQLite in Rails
presented_at
Talk delivered at wroclove.rb 2025 on 2025-03-14.
talk From PostgreSQL to SQLite in Rails
related_to
Wrona explicitly credits Stephen Margheim's wroclove.rb 2024 talk as the inspiration for his migration.

Provenance

Created
2026-04-17 16:18 seed
Read by
30 extractions