Skip to content

AI Code Tutorial · No. 01

Rotate database passwords on Railway, without restarting the database.

A brief you can hand to a coding agent. It changes the Postgres and Redis passwords behind a Flask app through a private tunnel, keeps both databases running, and tests every step before it moves on.

14 min read Intermediate
RailwayPostgreSQLRedisFlask
See the workflow

Why this, and why now.

On 21 October 2026 Railway disclosed that an unauthorised party may have reached the systems that store customer environment variables, and for some of them obtained the decrypted values. Its advice is plain: roll every variable stored on the platform. Read Railway’s security disclosure.

Most secrets are easy. You rotate an API key in the provider’s dashboard and paste the new one back. Database passwords are the awkward ones, because each one lives in two places at once: in the variable your app reads when it starts, and inside the database itself, which is what actually checks it.

Railway’s own guides use a Regenerate button that changes both in one step. It is the right call if you think a credential is being abused right now. It also restarts the database, drops every open session and leaves your apps unable to connect until you redeploy them.

This tutorial takes the other route. Your coding agent changes each half separately, and times the switch to the moment the new app container starts. The database never restarts. The window in which an app and its database disagree shrinks from minutes to seconds.

The idea

One password, two places.

The agent works from your machine. It reaches the databases through a private tunnel, and it changes the Railway variables through the CLI. Neither database is exposed to the internet at any point.

Fig. 1 — Who talks to what during a rotation
Architecture of an agent-led password rotation on Railway On the left, your machine runs the coding agent, which you supervise, and holds the new secrets in a private scratch folder. A private SSH tunnel created by railway connect leads into the Railway project's private network, where Postgres and Redis run. The agent also sets Railway variables through the CLI. The Flask app reads those variables only when a container starts, and connects to Postgres and Redis over the private network. YOUR MACHINE You approve each command Coding agent CLI · psql · redis-cli Scratch folder new secrets · mode 600 railway variable set --skip-deploys · over HTTPS railway connect --tunnel-only private SSH tunnel lands on 127.0.0.1 RAILWAY · PRIVATE NETWORK Variables POSTGRES_PASSWORD · REDIS_PASSWORD read once, at start Flask app gunicorn workers · Celery Postgres password inside the catalogue Redis requirepass in memory Architecture of an agent-led password rotation on Railway YOUR MACHINE You approve each step Coding agent CLI · psql · redis-cli Scratch folder · new secrets · mode 600 railway connect private SSH tunnel variable set --skip-deploys RAILWAY · PRIVATE NETWORK Variables POSTGRES_PASSWORD · REDIS_PASSWORD Flask app gunicorn · Celery · reads vars at start Postgres password in the catalogue Redis requirepass in memory

The variable.

What your app reads. Railway hands variables to a container once, when it starts. Changing a variable does nothing to a running app; only a new deployment picks it up. With --skip-deploys you can change it without restarting anything.

The database.

What checks the password. Postgres stores it in its own catalogue; Redis holds it in memory. Both accept a new password while running, and both keep already-open connections alive when it changes.

The whole method in one sentence: change the variable first without deploying, then flip the database at the exact moment the new app container boots.

The workflow

Six phases, each with a gate.

The agent does not move to the next phase until the current one passes its gate. Phases 0 to 2 change nothing your users can feel. Only phase 3 touches live traffic, and only after you say go.

Fig. 2 — The rotation, phase by phase

  1. PHASE 0

    Inspect

    Read-only. Map who uses which password, and check what each database really holds.

    Gate: Every consumer known

  2. PHASE 1

    Generate

    New secrets in private files. Postgres hash computed locally.

    Gate: Nothing printed

  3. PHASE 2

    Stage

    New values set on Railway without deploying anything.

    Gate: References resolve

  4. PHASE 3

    Redeploy & switch

    Redeploy the app. Flip the password as its new container starts.

    Gate: You said go

  5. PHASE 4

    Verify

    Logs, connections, old password refused, real pages served.

    Gate: All tests pass

  6. PHASE 5

    Clean up

    Delete secrets, close tunnels, write the report.

    Gate: Nothing left running

The switch window

Seconds, not minutes.

The order is what makes this safe. The agent switches the database only once the new container exists, so a build that hangs, fails or queues for twenty minutes changes nothing at all.

Fig. 3 — What each half of the password is doing, over time

  1. Stage

    The variable now holds the new password. Nothing restarts; the running app still uses the old one.

  2. Redeploy

    A new container builds. The old one keeps serving. This can take minutes, and nothing has changed yet.

  3. Switch · about 1–5 s

    The new container starts. The agent changes the database password within a second or two.

  4. Live

    Traffic moves to the new container, which already has the new password.

One Redis, several apps.

When two apps share a Redis, switch it as soon as the first new container starts, not the last. Otherwise that first app boots with the new password while Redis still expects the old one, and runs on whatever its fallback does until the switch.

Before you start

Ten minutes of preparation.

Railway CLI, logged in

The agent uses it for tunnels, variables, redeploys and logs. Check with railway whoami.

An SSH key on your Railway account

The private tunnel rides on it. Remember that it opens tunnels to every project you own.

psql and redis-cli

On the machine the agent runs on. The psql major version does not need to match the server for these queries.

Builds that work

Check the Railway status page. During the incident builds were paused; a rotation that cannot deploy strands your app.

A recent backup

Nothing here touches your data, but this is production. Take one anyway.

Approve every command

Run the agent in a mode where you confirm each command. If your tool has an automatic mode, switch it off for this job.

Phase by phase

What the agent actually does.

You do not need to type any of this. It is here so you can follow along and know what a good run looks like.

Phase 0 · read-only

Inspect before touching.

  • Map every consumer. Which services read which password, and whether as a reference like ${{Postgres.PGPASSWORD}} or as a pasted copy. Pasted copies do not follow a change; they get converted to references.
  • Check for public access. A public TCP proxy means the database is reachable from the internet. If nothing outside the project needs it, switch it off first.
  • Prove which password the database holds. Variables drift. The agent reads the stored SCRAM hash and checks each candidate variable against it locally, printing only true or false.
  • Check logging. log_statement should be none, so nothing about the change lands in server logs.
  • Know Redis. Uptime, persistence, users and connected clients. A Redis that has been up for a year with append-only persistence off is one you really do not want to restart.
  • Read the app’s start-up code. Does it retry the database on boot? Does its “Redis unavailable” fallback actually work with the library versions installed today?
# a private tunnel per database (discard output: it can echo secrets)
railway connect "Postgres" --tunnel-only -P 55432 >/dev/null 2>&1 &
railway connect "Redis"    --tunnel-only -P 56379 >/dev/null 2>&1 &

# which variables does the stored hash match? (true/false only)
psql "host=127.0.0.1 port=55432 user=postgres" -XAt \
  -c "select rolpassword from pg_authid where rolname='postgres'"

# logging, roles, who is connected
psql ... -c "show log_statement" -c "show password_encryption"
psql ... -c "select usename, client_addr, count(*)
             from pg_stat_activity group by 1,2"

# the tunnel lands on 127.0.0.1: a wrong password still connects
PGPASSWORD=wrong psql "host=127.0.0.1 port=55432 user=postgres" -c "select 1"

redis-cli -h 127.0.0.1 -p 56379 INFO server | grep uptime
redis-cli -h 127.0.0.1 -p 56379 CLIENT LIST

Gate: the agent reports every consumer, every mismatch and a plan per app, then waits.

Phase 1

Generate, never print.

  • 40 random letters and digits per database, from a cryptographic generator. Long, and nothing that needs escaping in a URL.
  • Private files, not the screen. A scratch folder outside the repository, readable by you only. Secrets move through files and standard input.
  • Keep the old Redis password. You need it to authenticate the change. Save it before staging anything.
  • Hash the Postgres password locally. The agent computes the SCRAM-SHA-256 verifier on your machine and sends only that. The plain password never reaches the server.
# the verifier Postgres stores, computed locally
import base64, hashlib, hmac, os
def scram(pw, it=4096):
    salt = os.urandom(16)
    sp = hashlib.pbkdf2_hmac("sha256", pw.encode(), salt, it)
    ck = hmac.new(sp, b"Client Key", "sha256").digest()
    sk = hmac.new(sp, b"Server Key", "sha256").digest()
    b = lambda x: base64.b64encode(x).decode()
    return (f"SCRAM-SHA-256${it}:{b(salt)}"
            f"${b(hashlib.sha256(ck).digest())}:{b(sk)}")

# later: ALTER ROLE postgres PASSWORD '<verifier>';

Gate: each verifier checks out against its password, and nothing secret appeared in the output.

Phase 2

Stage without deploying.

  • --skip-deploys is the whole trick. Without it, Railway redeploys the database and you are back to a restart.
  • Fix pasted copies now. An app holding a literal password gets a reference instead, so it can never drift again.
  • Check by comparison, not by looking. The agent confirms each app’s resolved variable equals the new secret, and that no deployment started.
# new value on the database service, no redeploy
railway variable set POSTGRES_PASSWORD --stdin --skip-deploys \
  -s "Postgres" < scratch/pg_new

railway variable set REDIS_PASSWORD --stdin --skip-deploys \
  -s "Redis" < scratch/redis_new

# a pasted copy becomes a reference
printf '%s' '${{Postgres.PGPASSWORD}}' | \
  railway variable set PGPASSWORD --stdin --skip-deploys -s "web"

Gate: every reference resolves to the new value and nothing has redeployed. Then the agent stops and asks you.

Phase 3 · live traffic

Redeploy, then switch on cue.

  • Redeploy the app and poll its deployment status every three seconds.
  • On DEPLOYING, switch. ALTER ROLE with the verifier for Postgres; CONFIG SET requirepass for Redis, authenticated with the saved old password. Open connections stay logged in.
  • Build failed or stuck? Do not switch. The old container keeps working on the old password. Nothing is broken.
  • Keep watching after success. A container can report success and crash twenty seconds later. The agent watches for a minute.
railway redeploy -s "web" -y

# poll until the new deployment leaves BUILDING
railway deployment list -s "web" --json   # every 3 s

# the moment it reads DEPLOYING:
psql "host=127.0.0.1 port=55432 user=postgres" \
  -v ON_ERROR_STOP=1 -f scratch/alter.sql
# alter.sql: ALTER ROLE postgres PASSWORD 'SCRAM-SHA-256$...';

printf 'CONFIG SET requirepass %s\n' "$(cat scratch/redis_new)" | \
  REDISCLI_AUTH="$(cat scratch/redis_old)" \
  redis-cli -h 127.0.0.1 -p 56379 --no-auth-warning

Gate: the stored hash equals the new verifier, Redis answers the new password, and the deployment is still healthy a minute later.

Phase 4

Verify like a sceptic.

A homepage returning 200 proves nothing if a CDN cached it. The agent requests a page that needs the database, with a cache-busting query string, reads the new deployment’s logs for authentication errors, and confirms the old password is refused. The full list is in the table below.

Phase 5

Leave nothing behind.

Delete the scratch secrets and scripts. Close every tunnel, and check the ports: stopping the CLI can leave the underlying SSH process listening. Finish with a short written report of what changed, every check and its result, and any downtime with timestamps.

The checklist

Every test the agent runs.

Twelve checks. None of them prints a secret; every comparison is true or false.

WhenTestHowPass
Before Stored password matches the variables Check the SCRAM hash in pg_authid against each candidate variable, locally Each app’s copy matches
Before Statement logging is off show log_statement none
Before Tunnel uses local trust Connect through the tunnel with a deliberately wrong password Connects
Before Builds are working Status page, and a recent successful deployment Green
Stage References resolve to the new secret Compare each app’s resolved variable with the new secret All true, no deploy started
Switch New hash is stored rolpassword equals the new verifier true
Switch Redis takes the new password PING with the new password PONG
Switch Redis refuses the old one PING with the old password WRONGPASS
After Deployment stays healthy Deployment status, one minute after success Still SUCCESS
After No authentication errors Logs: password authentication failed, WRONGPASS, NOAUTH, Traceback None after the switch
After The app really uses the database Uncached request to a database-backed page 200
After Nothing left running Scratch folder and tunnel ports Empty and closed

Failure modes

What can go wrong, and the fix.

The build hangs.

The agent waits and switches nothing. The old container keeps serving on the old password. Retry the build later.

The new container crashes after the switch.

Read its logs. Both halves of the password now match, so a fresh redeploy usually comes up clean.

A fallback path is broken.

Code that only runs when Redis is down has often never run at all. Read it in phase 0, and pin the library it depends on.

A password was pasted, not referenced.

That app keeps the old value and fails after the switch. Phase 0 finds these; phase 2 converts them.

A tunnel outlives the session.

Stopping the CLI can leave its SSH process holding the port. Check with lsof and close it.

A secret gets printed.

Treat it as exposed and rotate that one again. The prompt forbids printing, but tools can echo connection strings.

The prompt

Hand this to your coding agent.

Fill in the parts in square brackets. The agent inspects first, reports, and waits for your go-ahead before it touches live traffic. 

You are going to rotate the database passwords behind a web app hosted on Railway.

Project: [Railway project name], environment: [production]
Databases: [Postgres service name(s)], [Redis service name]
Apps that use them: [app service name(s)]

Goal: a new password for every database, with no database restart and no more than a few seconds in which an app and its database disagree.

HARD RULES
- Never print, log or paste a secret. Keep secrets in files with mode 600 in a scratch folder outside the repository. Pass them through files or standard input. Compare values with equality or hash checks only.
- Never restart, redeploy or "Regenerate" a database service. Use --skip-deploys on every variable change.
- Work read-only until I approve. Stop and wait for me after Phase 0 and after Phase 2, and whenever a check fails.
- If a command is denied, do not look for a workaround. Tell me.

PHASE 0 - INSPECT (read-only)
1. Confirm the Railway CLI is logged in and linked to the right project and environment. Confirm builds are working: a recent successful deployment, and the Railway status page.
2. For each database, list every service that uses its credentials and whether each one uses a reference (${{Service.VAR}}) or a pasted literal. Report every literal.
3. Report any database with a public TCP proxy. Do not change it.
4. Open one tunnel per database: railway connect "<service>" --tunnel-only -P <port>. Discard its output; it can contain secrets.
5. Postgres: read the stored SCRAM verifier from pg_authid and check locally which variables it matches (POSTGRES_PASSWORD, PGPASSWORD and each app's copy). Report the server version, log_statement, password_encryption, login roles and current connections.
6. Connect through the tunnel with a deliberately wrong password. If it does not connect (no local trust), stop: we must save the old password before staging.
7. Redis: PING with the current password. Report the version, uptime, persistence settings, ACL users (hashes redacted) and connected clients.
8. Read the app's start-up code. Does it retry the database on boot? What does it do if Redis rejects it, and does that fallback work with the library versions installed in production?
Then report your findings and a plan per app, and wait.

PHASE 1 - GENERATE
9. Generate a 40-character alphanumeric password per database with a cryptographically secure generator, into mode-600 files.
10. Save the current Redis password to a file before changing anything.
11. For Postgres, compute the SCRAM-SHA-256 verifier locally (4096 iterations, random 16-byte salt) and check it against the new password.

PHASE 2 - STAGE (no restarts)
12. Set the new POSTGRES_PASSWORD and REDIS_PASSWORD on the database services with: railway variable set <VAR> --stdin --skip-deploys -s "<service>"
13. Replace every pasted literal in an app with a reference to the database service's variable.
14. Confirm each app's resolved variable now equals the new secret, and that no deployment started.
Then show me the exact plan for Phase 3, and wait.

PHASE 3 - REDEPLOY AND SWITCH (only after my go-ahead)
15. Redeploy one app, or one batch. Poll railway deployment list --json every 3 seconds.
16. The moment its new deployment reaches DEPLOYING, run ALTER ROLE <user> PASSWORD '<verifier>' through the tunnel. Then confirm the stored hash equals the new verifier.
17. For a Redis shared by several apps, run CONFIG SET requirepass (authenticated with the saved old password, value sent over standard input) as soon as the FIRST consumer's new container starts, not the last.
18. If a deployment fails or hangs, do not switch that database. Report and wait.
19. Watch each new deployment for 60 seconds after it reports SUCCESS. If it crashes, read its logs, report, and redeploy once the cause is understood.

PHASE 4 - VERIFY
20. Each new deployment still reports SUCCESS after 60 seconds.
21. The new deployments' logs show no "password authentication failed", WRONGPASS, NOAUTH or tracebacks after the switch.
22. Redis: the old password gets WRONGPASS; the new one gets PONG.
23. pg_stat_activity and CLIENT LIST show connections from the new containers.
24. A database-backed page returns 200 with a cache-busting query string, so no CDN cache answers for it.

PHASE 5 - CLEAN UP
25. Delete the scratch secrets and scripts.
26. Close every tunnel, and check with lsof that no SSH process still listens on the tunnel ports.
27. Give me a short report: what changed, each check and its result, any downtime with timestamps, and anything left for me (local .env files, external tools that used the old passwords).

Works with any agent that can run shell commands: Claude Code, Codex, Cursor and the like. Run it with command approval switched on.

Questions

The short answers.

Does changing a Postgres password on Railway need a database restart?

No. PostgreSQL keeps role passwords in its own catalogue, so ALTER ROLE takes effect immediately and the server keeps running. The POSTGRES_PASSWORD variable is only read when the data directory is first created. Railway’s Regenerate button restarts the database to apply a new password; the method in this tutorial does not.

Do open connections drop when the password changes?

No. In both PostgreSQL and Redis, sessions that are already authenticated stay authenticated. Only new connections need the new password. That is why the switch can be timed to the moment a new app container starts.

Why not just press Regenerate in the Railway dashboard?

It is simpler and it is Railway’s documented route, so use it if you believe a credential is being abused right now. The cost is a database restart, dropped sessions and an outage until every dependent service has been redeployed. This tutorial trades that for a few more steps and an agent doing them carefully.

Is it safe to let an AI agent touch production database credentials?

It can be, with rules. The agent never prints a secret, works read-only until you approve, stops at fixed gates, and runs in a permission mode where you approve each command. Treat it like a careful junior engineer with a checklist, not an autopilot.

What happens if the app build fails halfway through?

Nothing breaks. The database password is only switched once the new container is starting, so a failed or stuck build leaves the old container running on the old password. You retry the build, then switch.

Does this only work for Flask?

No. Any app that reads its database and Redis credentials from environment variables at start-up behaves the same way: Django, FastAPI, Rails, Node, Celery workers. The example here is a Flask app served by gunicorn because that is a common Railway setup.

Sources

Keep going

All AI Code Tutorials Briefs you can hand to a coding agent, tested on real systems.