Railway security incident: exposed environment variables and the risk to AI API keys

On October 21, 2026, the cloud application platform Railway disclosed a security incident in the internal systems that store customer environment variables. The company detected anomalous activity and said an unauthorized third party may have accessed encrypted variables and, worse, may have obtained the decrypted contents of a subset of them. Railway also assessed that the event could be part of a broader campaign against platform companies, with the goal of compromising AI inference providers.

For engineering and security teams, the disclosure separates two things that are easy to blur: a cryptographic failure and an operational compromise. Nothing in it suggests the algorithms protecting the data at rest were broken, or that key management was subverted at the algorithmic level. What it exposes is how fragile secret management becomes when the path that decrypts and delivers secrets may have been infiltrated.

Environment variables carry the credentials a deployed service uses to authenticate against databases, object storage, payment gateways, and increasingly, expensive AI inference APIs. A platform that stores them for thousands of tenants becomes a high-value concentration point. Attackers going after those platforms want more than source code or personal data. They want API keys that give immediate, monetizable access to external compute.

What happened in the Railway incident

Railway’s status page showed degraded availability or outright outages across the dashboard, public API, command-line interface, and support platforms. The company paused builds and deployments in all regions, which froze the state of the platform and protected forensic evidence. It also created a problem for customers: the deployment pipeline is the mechanism most of them would use to push rotated credentials, and it was switched off.

Restoration followed a staged order. Enterprise workspaces came back first, then Pro, Hobby, and Free. The reasoning is sound from a capacity standpoint, since it keeps build clusters from being swamped by a wave of simultaneous redeployments. For security remediation it was painful. Customers on the lower tiers knew their variables might be compromised but could not ship new credentials to their running containers while the build system stayed offline.

Railway’s containment: token resets and credential rotation

Railway invalidated all Railway API tokens and OAuth sessions, citing impacts on adjacent cache systems. It also rotated its own infrastructure credentials: the S3-compatible access credentials for its storage buckets were invalidated and regenerated, and GitHub integration tokens were severed. So that workspace administrators could tell company-mandated resets from malicious activity in their audit logs, Railway named internal engineers, including Faraz Patankar, as the people who executed those resets.

This follows a basic rule of incident response. When you cannot yet size an intrusion into a shared secrets system, you assume every adjacent authentication path is hostile and cut it.

How Railway’s envelope encryption works

To judge the severity, you need to know how Railway protects customer secrets. In April 2024, Railway published an engineering retrospective describing its migration of more than three million environment variables to envelope encryption. Before that, the platform called Google Cloud Key Management Service synchronously for every encryption and decryption operation. That was cryptographically sound, but it added latency and ran into API quotas, especially for microservice architectures that resolve hundreds of variables in one deployment.

The two-tier key hierarchy

Envelope encryption removes that bottleneck with two layers of keys: Data Encryption Keys (DEKs) and Key Encryption Keys (KEKs). The platform generates a unique 256-bit symmetric DEK locally for a given environment, such as production or staging. The DEK encrypts the environment variables with AES-256-GCM. The DEK is then encrypted, or wrapped, by a KEK that stays inside the Google Cloud KMS boundary. The database stores the encrypted variable next to the wrapped DEK, so the plaintext key never lands on disk.

When a component needs a variable’s plaintext value, it fetches the encrypted payload and the wrapped key, asks KMS to unwrap the key, and decrypts the variable in memory with the resulting DEK.

Why AES-256-GCM matters here

Galois/Counter Mode (GCM) provides confidentiality and authenticity as an Authenticated Encryption with Associated Data (AEAD) algorithm. Any change to the ciphertext or its associated metadata makes tag verification fail, so tampered data is never decrypted.

Why encryption at rest did not protect the secrets

Railway has not published a root cause in the details I have, so what follows is my reading and not a confirmed finding. The weak point appears to sit outside the mathematical guarantees of AES-256-GCM. The cipher did its job on the data stored in the database. If the compromise happened, it happened in the execution environment or the authorization path that requests the key unwrap.

An attacker who can run code inside the service that performs KMS unwrapping, or who steals the short-lived service account credentials allowed to make those calls, never has to break the cipher. They ask the authorized system to decrypt the data for them.

GCM nonce reuse

GCM also brings implementation risks of its own. It needs a unique nonce (initialization vector) for every encryption under a given DEK. If a bug reuses a nonce with the same key, confidentiality and authenticity collapse, and an attacker can potentially recover the authentication key and forge valid ciphertexts. This is a general risk of the mode and nothing in the disclosure says it occurred here.

Plaintext keys in server memory

Once KMS returns the unwrapped DEK, it sits in plaintext in the host’s memory. An attacker with remote code execution, or enough privilege to dump process memory, can scrape the heap and pull out decrypted environment variables without touching the database encryption at all. Encryption at rest is a baseline compliance requirement. The real perimeter is the set of identity and access management policies that decide which workloads, under which conditions, may materialize plaintext secrets in memory.

Why attackers target AI inference API keys

Railway’s assessment that the incident belongs to a campaign against AI inference providers reflects a shift in what attackers want. Over the past three years, large language models have become a standard part of application architectures, and that has created a liquid market for stolen API credentials. A traditional breach steals a database of user records to sell on dark web forums. Stolen inference keys pay off immediately, as compute.

Providers such as OpenAI, Anthropic, OpenRouter, and Grok bill by token volume, model complexity, and compute time, and often require substantial prepaid balances or credit lines. An attacker who breaches a platform hosting thousands of AI-driven applications does not have to reverse-engineer a model or steal training data. They harvest the environment variables that hold the provider keys. A single successful intrusion into the secret management layer could yield hundreds of valid, high-limit keys. Those keys can run unauthorized compute-heavy workloads, drain prepaid balances, or spread malicious traffic across a pool of stolen credentials to get around rate limits.

Model hijacking and account bans

The direct financial damage can be severe for startups and enterprises on tight compute budgets, but there is more. Threat actors use stolen keys to query the provider for malicious purposes: writing phishing payloads, automating the search for zero-day vulnerabilities in target software, or producing deepfake assets for social engineering. Because the requests carry a valid key belonging to a legitimate business, the provider’s abuse detection may not flag them at first, and the traffic gets attributed to the victim. The compromised company then risks a permanent ban from the provider for terms-of-service violations, which can knock out its own production applications.

The platform becomes an aggregator that concentrates the credentials for dozens of high-value third-party services in one control plane. That is why the return on compromising a shared secret management system is so much higher than phishing individual developers.

How to audit OpenAI and Anthropic logs for stolen API key abuse

On October 22, Railway published provider-specific remediation guides for Stripe, AWS, GCP, GitHub, PostgreSQL, MySQL, and the major AI inference providers. When you follow its advice to audit provider-side logs, look past aggregate token counts. In platforms like OpenAI and Anthropic, check these signals:

• Model selection. Look for sudden shifts toward older, less restricted, or specialized models your application does not normally call.
• Prompt structure. Look for repetitive, automated patterns that suggest bulk data generation or automated vulnerability scanning.
• Request origin. If your application serves a domestic user base but the logs show authentication from scattered international IP addresses or known cloud hosting ranges, treat that as a near-certain sign of credential theft.

The credential rotation deadlock after Railway paused deployments

Credential rotation looks atomic in theory: revoke the compromised credential, issue a new one. In a containerized, continuously deployed system it is a multi-stage process involving the secret issuer, the configuration platform, and the running application. The old credential is invalidated at the source, the new one is injected into configuration, the application is reloaded or redeployed to pick it up, and the system verifies that the new credential works.

Pausing builds and deployments broke that sequence and left customers in a deadlock. Revoke an OpenAI or AWS key at the provider right away, and the running containers still holding the old key in memory start failing authentication, which means a full outage. Wait for Railway’s build system to return so you can redeploy with new variables, and the attacker keeps a window to abuse the live credentials.

Customers using Railway’s managed storage buckets and GitHub integrations felt this directly. Railway invalidated and regenerated the S3-compatible credentials and GitHub tokens on its backend to protect its own infrastructure. Running containers had loaded the old tokens at startup, so those applications immediately lost access to storage and source repositories.

Rotating PostgreSQL and MySQL credentials without locking yourself out

Railway’s PostgreSQL and MySQL rotation guides touch the hardest case, stateful services. A stateless web server can be terminated and replaced by an instance with updated variables. Database connections usually sit in a pooling library that keeps long-lived TCP connections open. When you rotate a database password at the source, pooled connections do not re-authenticate. They keep running on their cached, authenticated sessions until the server drops them or the pool cycles.

That has two consequences. If an attacker already established a foothold with the compromised credentials, changing the password in Railway variables and redeploying the application will not necessarily evict them, because their sessions are independent and persistent. And during rollout, some pods authenticate with the new password while others still use the old, revoked one. In that split state the database may lock the account after too many failed attempts, a self-inflicted denial of service on top of the original incident.

Mature teams handle this with overlapping credential lifecycles. Both the old and new database passwords stay valid for a short window, so connection pools can drain and reconnect with the new secret before the old one is destroyed. Railway’s build pause prevented customers from running that kind of staged rotation, which shows the systemic risk of coupling the deployment pipeline tightly to secret management.

Tenant isolation: why cryptographic separation is not enough

The incident also shows the limits of relying on cryptography alone to separate tenants. A unique DEK per workspace or environment means one compromised key does not mathematically expose another tenant’s ciphertext. It does nothing against a compromise of the central service that manages those keys. If the internal agent that brokers decryption requests runs with overly broad privileges, or the logic that maps a requesting container to its DEK has a flaw, an attacker who takes over that broker can cross tenant boundaries.

Multi-tenant security needs isolation at the identity and access management layer. Every DEK unwrap request should carry a narrowly scoped, cryptographically verifiable workload identity and be checked against a policy that explicitly denies access to other tenants’ secrets. Audit logs for decryption operations should be immutable, granular, and separated from the systems that perform decryption, so an attacker who controls the secret broker cannot alter or delete the record of their own access.

Zero-trust secret delivery with SPIFFE

The common model has a central platform agent inject environment variables into a container at startup from a static configuration. It depends entirely on the security of the orchestration layer. If that layer is compromised, the attacker can inject malicious variables or exfiltrate the real ones before the application starts.

The industry is moving toward frameworks like SPIFFE (Secure Production Identity Framework for Everyone), which gives every workload a cryptographic identity on Kubernetes, virtual machines, or serverless platforms. Under SPIFFE, an application does not rely on pre-injected environment variables for its most critical secrets. It uses a short-lived, signed SVID (SPIFFE Verifiable Identity Document) to mutually authenticate with a dedicated secrets broker or service mesh. The broker checks the workload’s identity, its runtime context, and the authorization policy, then streams the credentials directly into the application’s memory. The operating system’s environment variable block is bypassed. An attacker who reads the container environment or the host’s configuration files finds nothing useful, and the real secrets are rotated continuously in the background.

Least privilege for API keys, database credentials, and cloud keys

Apply least privilege to the secrets themselves. An AI inference key should be limited to the models and endpoints the application needs, so a stolen key cannot reach organization settings or fine-tuning endpoints. Database credentials should be scoped to specific schemas and, where possible, read-only roles. Cloud provider keys should be restricted through granular identity and access management policies.

Key takeaways from the Railway breach

• Encryption at rest protects stored ciphertext. It does not protect secrets from a compromised path that is authorized to decrypt them.
• The platform’s own containment steps, including the build pause, can block customers from rotating credentials safely, so plan rotation procedures that do not depend on a single deployment pipeline.
• AI inference keys are now a prime target because they convert directly into compute. Audit provider logs for unusual models, prompt patterns, and request origins.
• Database rotation needs overlapping old and new credentials, and live pooled sessions have to be accounted for.
• As more architectures consolidate on centralized deployment platforms, the plaintext materialization of secrets deserves the same priority as the encryption protecting them: workload identity, ephemeral credentials, and strict tenant isolation.