Rotating a credential sounds like a mechanical task: generate a new one, replace the old one, done. In an architecture with a handful of services it's almost that simple. The problem appears when that same key is copied in six different places, used by a batch process that only starts up in the early hours, and nobody remembers whether there's a scheduled task that still points to the previous version. At that point, rotation stops being a value change and becomes a project requiring order, verification, and — if something goes wrong — a service outage nobody saw coming.
What follows is the procedure we apply when planning credential rotation in systems with several interdependent services, without going into any specific provider or tool, because the pattern applies equally to a database, an external API, or an internal certificate.
The inventory almost nobody keeps up to date
Before touching anything, you need to answer two questions for each credential: who uses it, and where is it copied. They look like the same question, but they aren't. A service can use a credential without anyone having it saved in its configuration, because it inherits it from an environment variable that in turn comes from another system. And a credential can be copied in a deployment script, in a scheduled task, on the laptop of someone who put it in a local file to test something months ago and never deleted it.
The inventory has to capture both things separately. For each credential we note the service or environment it belongs to, the list of consumers that use it at runtime, and the list of places where it's stored in plain text or encrypted: configuration files, secrets managers, environment variables, infrastructure scripts, even internal documentation that sometimes includes examples with real values by mistake.
A simple schema, sufficient to get started, might look like this:
credencial: db-pedidos-produccion
tipo: contrasena-base-datos
consumidores:
- servicio-api-pedidos
- job-conciliacion-nocturna
- script-backup-diario
almacenada_en:
- gestor-secretos/produccion/db-pedidos
- variable-entorno: DB_PEDIDOS_PASS (servicio-api-pedidos)
- fichero-config: /etc/batch/conciliacion.confWithout this map, the dual-credential pattern that comes next turns into a lottery: you can deploy the new key in the three places you remember and unknowingly leave a fourth copy alive, which will keep authenticating with the old one until you revoke it and something stops working without warning.
Order matters: first what others need
Once the inventory is in place, it's time to decide the rotation order. The rule is easy to state and easy to skip under pressure: rotate first what other services need to start up, and leave for last what depends on them.
Imagine an architecture with a central authentication service, a business API that validates tokens against that service, and an admin panel that consumes the API. If you rotate the panel's credential first, nothing serious happens if something fails: the impact stays with a single consumer. But if you rotate the authentication service's credential first without having verified that all its consumers support the change, you can leave the entire chain that depends on it unable to start up.
That's why the practical order is usually:
- •Base infrastructure credentials (databases, message queues, storage) used by several services.
- •Credentials of intermediate services consumed by others (authentication, internal gateways).
- •Credentials of user-facing or one-off services, which usually have fewer dependents.
The risk still exists, but this way, if something fails, it fails at the point where the blast radius is smaller and easier to diagnose.
Create, deploy, verify, and only then retire
The core pattern of any rotation without downtime has four steps, and the order is not negotiable:
Create the new credential without touching the old one
The new credential coexists with the old one from the very first moment. This requires that the service or provider issuing the credential allow having two valid ones at the same time, even if only for a short window. If it doesn't allow that, the pattern changes, and we cover that in the next section.
Deploy the new one to each consumer in the inventory
At this step, the inventory from the first section becomes a task list. Each identified consumer receives the new credential, one by one, without rushing. This is when you discover forgotten copies: if a consumer keeps failing after being updated, there's probably a second copy of the old credential somewhere the inventory didn't capture.
Verify that the new one works at each point
Deploying the credential doesn't guarantee it works, so each point is verified. For a database credential, for example, a minimal check is forcing a test connection with the new credentials and confirming a valid response:
# comprobación genérica de conexión con la credencial nueva
if ! check-connection --cred-file nueva-credencial.env; then
echo "La credencial nueva no funciona en este consumidor, no retirar la vieja"
exit 1
fi
echo "Conexión verificada con la credencial nueva"The goal of this step is to have explicit proof, not an assumption, that each consumer is already using the new credential and working with it.
Retire the old one and verify that it actually fails
Only once all consumers have confirmed the change is the old credential revoked. And here there's a step that gets forgotten frequently: after revoking it, you have to check that it actually stops working. If the old credential still authenticates after being retired, something wasn't revoked where it should have been, and that's just as serious as the opposite failure.
When the provider gives no room to coexist
Not all systems allow two active credentials at once. Some providers invalidate the previous one the very instant the new one is generated, with no overlap window. In that case the dual-credential pattern isn't possible as such, and you need to plan an explicit downtime window: notice is given in advance, a low-usage time slot is chosen, the new credential is deployed to all consumers simultaneously or in as short a sequence as possible, and it's accepted that during that interval some services will fail until they receive the updated value.
The difference from an improvised rotation is that this window is decided and communicated beforehand, and it has a time limit. In an improvised rotation, the outage appears because someone didn't realize the provider invalidated the previous key immediately.
One credential per service and per environment
A good part of complicated rotations stem from an earlier mistake: reusing the same credential across several services or several environments to save configuration work. If the production credential is the same as the testing one, rotating it in production forces you to also touch the testing environment, and vice versa. If the same value is used in two different services, the consumer inventory doubles and the risk of leaving a copy unupdated multiplies.
Segregation is the cheapest preventive measure mentioned in this article: each service and each environment with its own credential, with no repeated values between them. It doesn't eliminate the need to rotate, but it drastically reduces the number of consumers affected by each rotation and makes the inventory much easier to keep up to date.
Leaving a trail at every step
In a process of compliance with the National Security Framework (ENS), besides doing credential rotation properly, you need to be able to prove how it was done. That means logging, for each rotation, which credential was rotated, on what date, which consumers received it, how it was verified to work at each one, and when the previous one was revoked, along with confirmation that it actually stopped being accepted.
A minimal log, for example in the form of a record per rotation, might capture this:
Credencial rotada: db-pedidos-produccion
Fecha de creación de la nueva: 2024-XX-XX
Consumidores desplegados y verificados:
- servicio-api-pedidos (verificado 2024-XX-XX, hora)
- job-conciliacion-nocturna (verificado 2024-XX-XX, hora)
- script-backup-diario (verificado 2024-XX-XX, hora)
Fecha de revocación de la vieja: 2024-XX-XX
Comprobación de revocación: intento de conexión con credencial antigua, resultado rechazadoThis record, accumulated rotation after rotation, is the evidence an ENS auditor asks to see when reviewing the key management control: it's not enough to say rotation happens periodically, you need to be able to show when, how, and with what result.
Errors that keep appearing again and again
Almost all rotation incidents that end in an outage share the same root cause: one of the above steps gets skipped, not some exotic technical failure. The most common ones:
- •Revoking the old credential before having verified that all consumers already use the new one, trusting that the deployment went well without checking it.
- •Not verifying that the old credential actually stops working after revoking it, leaving open access that nobody detects until someone takes advantage of it or it shows up in a review.
- •Rotating without a complete inventory, discovering hidden consumers mid-process, usually at the worst possible moment.
- •Reusing the same credential across environments, which turns a local rotation into one that affects systems that had no reason to be involved.
This pattern doesn't save you planning, but it gives it a fixed order. Before scheduling the next rotation, it's worth dedicating a session solely to updating the consumer inventory: it's the step that saves the most time when the moment comes to execute.