GitLab’s 2017 Database Incident: A Retrospective on Backups and Error Signals
Let's discuss your project →GitLab’s 2017 post-mortem is a useful reminder to check backups, notifications, and restore readiness: a historical case that can be turned into a practical exercise for a small team. GitLab, February 10, 2017
In the report published on February 10, 2017 about the January 31 incident, GitLab describes backups failing because of an incompatibility between pg_dump and the database version, error notifications being rejected, and recovery through an available snapshot. These are facts about the historical case, not a description of GitLab’s current infrastructure.
The operational question
Dedaloop’s editorial reading is this: who notices that a backup was not produced, and who checks that it can actually be used? Planning a job, receiving a completion signal, and restoring a service are three different checks.
There is no need to reproduce the incident to discuss them. The proposed lab uses a fictitious database and a checklist; it does not require deleting anything in production and it is not a service carried out for a client.
A three-step test
- Inventory: for the chosen service, list the backup tool, version, destination, frequency, and owner. If a field is unknown, record it as follow-up work.
- Notification: in a test environment, use the monitoring system’s built-in test mechanism to verify the recipient, fallback channel, and handoff. Do not deliberately block real backups.
- Recovery: restore a test dataset in a separate environment and validate known data. Note dependencies, required credentials, and steps that need a specific person.
Do not confuse the indicators
The age of the last valid backup, the last alert actually received, and the date of the last restore test answer different questions. Keeping them separate helps reveal a system that produces copies but has never been tested, or an alert sent to a recipient who is no longer available.
To close the session, assign an owner to every gap and decide when to repeat the test. A version upgrade is a good moment to check tool compatibility again, but it is not automatic proof that something will break.
The value of this story is not to blame a single error for all of an organization’s problems. It is to turn a public report into verifiable questions about your own service, while keeping GitLab’s documented experience distinct from the choices proposed here.