Case studies
What real upgrades turned up
Seven trial upgrades we ran on public open-source apps, with their own test suites. In every one, each test that passed before the upgrade still passed after it. In five, review still found problems the tests didn't catch.
These are trial upgrades, not paid orders. We ran them in October 2026 on public open-source projects to test our process before taking customers. The projects' maintainers didn't commission them, and apart from Node-RED, the changes haven't been submitted upstream. The project names link to their public code so you can see what we worked on.
The versions we upgraded are older releases, and most of these projects have since made the same upgrade themselves. Under each case we say what they did and how it compared with our result. Checked against each project's repository on 23 October 2026.
At a glance
| Project | Upgrade | Tests before → after | Problems tests missed |
|---|---|---|---|
| Baby Buddy | Django 4.2 → 5.2 LTS | 291 → 291 passing | 2 |
| Node-RED | Express 4 → 5 | 2,869 → 2,869 passing | 3 |
| Koel | Laravel 10 → 11 | 442 → 442 passing | 2 |
| Pinkary | Laravel 11 → 12 | 733 → 733 passing | 1 |
| json-server | Express 4 → 5 | 128 → 128 passing | 1 |
| healthchecks | Django 4.2 → 5.2 LTS | 1,457 → 1,457 passing | None |
| pretalx | Django 4.2 → 5.2 LTS | 1,388 → 1,388 passing | None |
Tests that failed before the upgrade for reasons outside the code (no internet, no IPv6 or missing system tools in our test environment) failed the same way after it and are left out of the counts.
Django
Baby Buddy v2.1.2
A self-hosted app parents use to track sleep, feeds and nappies.
- Upgrade
- Django 4.2 → 5.2 LTS, plus 12 add-on packages
- Tests before → after
- 291 → 291 passing
What broke
12 tests failed after the version bump, from changed test APIs and Django 5's removal of pytz time zone support. All fixed.
What the tests missed
- Uploads going to the wrong place. Django 5.1 ignores the old
DEFAULT_FILE_STORAGEandSTATICFILES_STORAGEsettings without raising an error. Sites using Amazon S3 would have quietly saved parents' photo uploads to the server's local disk, and production would have lost its compressed static files. Every test passed. - Wrong totals outside UTC. Per-user time zones relied on pytz, which Django 5 no longer handles. Sleep and feeding totals would have been wrong for every user outside UTC. Only 2 tests caught the symptom.
Decision the owner would make
Whether to move 12 add-on packages, several by a major version, in the same upgrade.
Baby Buddy's own Django 5 upgrade (pull request #746, January 2024) made the same storage-settings fix. The project now runs Django 5.2.
Node.js
Node-RED 5.0.8
A widely used low-code tool for wiring together devices, APIs and services.
- Upgrade
- Express 4 → 5
- Tests before → after
- 2,869 → 2,869 passing, plus 5 new tests
What broke
53 tests failed after the bump, from Express 5's new route syntax and changed internals. There was no upstream fix to compare against.
What the tests missed
- Deleted endpoints that keep answering. Express 5 renamed the internal router Node-RED uses to remove HTTP routes on redeploy. Without a fix, endpoints a user deleted would have kept responding.
- Different data in user flows. Express 5's default query parser no longer expands nested parameters such as
a[b]=c. Every flow reading nested query values would have silently received different data. We kept the old parser as the default. - Tests that weren't testing the upgrade. Node-RED's node test helper bundles its own copy of Express 4, so its node tests never ran against Express 5. We verified the change by running the real server end to end.
Decision the owner would make
Users' own HTTP In URLs written in the old syntax (such as /files/*) must be rewritten (/files/*path). The upgrade reports this clearly on the affected node instead of breaking the whole deploy.
After the Node-RED maintainers asked for it, this work was submitted to the project as pull request #6024, with a compatibility layer so existing community nodes keep working. As of 23 October 2026 it is open and waiting for review; Node-RED 5.0.9, released 21 October, still uses Express 4.
Laravel
Koel v7.5.2
A self-hosted music streaming server.
- Upgrade
- Laravel 10 → 11, PHP minimum 8.1 → 8.2, 64 packages
- Tests before → after
- 442 → 442 passing
What broke
Laravel 11 removed its Doctrine DBAL integration, and one add-on (a .env editor) never got a Laravel 11 release.
What the tests missed
- An install command that lies.
koel:init, the command every user runs to install and upgrade, called a removed Laravel method inside a catch-all error handler. It would have reported "cannot connect" on a perfectly healthy database. - Broken fresh installs. Fresh SQLite installs crashed on 2 migrations, and SQLite foreign keys came out duplicated and conflicting. Comparing the schema of a fresh install before and after the upgrade found differences no test checked.
Decision the owner would make
Raising the PHP minimum, and replacing the abandoned add-on with a maintained fork (checked to be code-identical).
Koel's own Laravel 11 upgrade (pull request #1960, June 2025) made the same koel:init fix and also swapped the abandoned add-on for a fork. The project now runs Laravel 13.
Laravel
Pinkary v1.44.0
An open-source social app built with Laravel.
- Upgrade
- Laravel 11 → 12, 63 packages
- Tests before → after
- 733 → 733 passing
What broke
Static analysis needed upgrading too: older Larastan crashed, and stricter Laravel types surfaced errors that were fixed with precise types rather than blanket ignores.
What the tests missed
- Sign-ups that would have stopped working. Moving the Cloudflare Turnstile package to 3.x removed a Blade directive the sign-up form used, and versions 3.0.0 to 3.0.3 also rejected every valid response. The upgrade stayed on Turnstile 2.1.
Decision the owner would make
Turnstile 2.x vs 3.x, a QR code library going from 2 to 3, and time-ordered UUIDs. Pinkary's own Laravel 12 upgrade (July 2025) made the same choices: Turnstile 2 and the QR code library at 3.
The Turnstile bug was fixed in version 3.0.4 (September 2026). Pinkary now runs Laravel 13 with Turnstile 3.
Node.js
json-server v0.17.4
A popular tool that turns a JSON file into a fake REST API for prototyping.
- Upgrade
- Express 4 → 5
- Tests before → after
- 128 → 128 passing
What broke
25 tests failed because Express 5 makes query strings read-only. The test runner had to move from Jest 26 to 29 as well.
What the tests missed
- Installs that break later instead of failing early. The package still declared support for Node.js 12, but Express 5 needs Node 18 or newer. Installs on older Node would have broken at runtime instead of being refused.
Decision the owner would make
Raising the minimum Node.js version from 12 to 18.
json-server's own version 1, in beta since January 2024, replaces Express altogether, so this upgrade applies only to its older 0.17 releases.
Django
healthchecks v3.0
A cron job and background task monitoring service.
- Upgrade
- Django 4.2 → 5.2 LTS
- Tests before → after
- 1,457 → 1,457 passing
What broke
2 old database migrations used an option Django removed; they were made no-ops. 14 type-checker errors fixed.
Decision the owner would make
Whether to keep the old "http" default for web addresses typed without a scheme ahead of Django 6.0. We kept it.
Our migration fix matched the project's own, made in June 2024 before its Django 5.1 upgrade. On the web-address default, the project later chose "https" (January 2025). It now runs Django 6.1.
Django
pretalx v2024.1.0
Conference planning software used to run calls for papers and schedules.
- Upgrade
- Django 4.2 → 5.2 LTS, plus 10 add-on packages
- Tests before → after
- 1,388 → 1,388 passing
What broke
611 tests failed, all from one changed internal Django method used by the plugin system. One fix. 2 new migrations were required.
Decision the owner would make
Dropping Python 3.9, which Django 5.2 no longer supports, and checking production for duplicate settings rows before the new migrations run.
Our two new migrations matched the two the project added itself in September 2025. It now runs Django 6.1.
What we took from them
- Passing tests is the floor, not the finish. In five of seven apps, the most serious problem was one no test caught. That's why we review every upgrade for settings the new version silently ignores, removed internals and changed defaults, rather than just running your tests.
- Every upgrade needs a decision only you can make. A minimum language version, an add-on's major version, a syntax change your users will notice. We flag these in your free scan report, before you pay.
- These apps had unusually good tests: 128 to 2,869 each. If yours has fewer, the review matters more, and the scan will tell you honestly what that means for risk and price.
Find out what your upgrade will take
A free report on what breaks, and a fixed price, within one business day. No sales call.
Get a free upgrade scan