Operating it

Most of this platform runs on request. What does not is a handful of scheduled jobs, and those are the part that can stop working without anybody noticing — which is what this page is about.

The scheduled jobs

The ordering is the load-bearing part of that table, and every gap in it is a decision.

Bandwidth runs at twenty past, not on the hour. CloudFront delivers access logs with a lag, and starting on the hour races the delivery.

Cleanup runs before the storage sample, not after. Swept objects have to be gone before the thing that measures them, or a customer is billed for uploads that were abandoned and deleted the same night.

auth-users runs after the purge and before the charge. After, so a project destroyed last night is not counted and billed on its way out. Before, so the number being charged was taken today rather than yesterday.

auth-cleanup is not the auth service. It sweeps the console's sign-in throttle and always has. The name predates there being an auth service and now reads as though it belonged to one; renaming it would change its cron path and orphan its run history, so the label carries the correction instead.

When a job fails

Every job runs through a wrapper that records the attempt, its duration and its outcome, then mails PLATFORM_OWNER_EMAILS.

1{ "ok": false, "job": "files-storage", "error": "The job failed. The detail was recorded." }

The response carries no detail on purpose — a job's error can contain a connection string, and this route is one misconfiguration away from being public. The detail is in the record and in the mail.

When a job doesn't run at all

This is the failure that alerting cannot catch. A job that never fires raises no error, sends no mail, and looks exactly like a quiet night — which is also the state after deploying a schedule that does not work.

So every job declares how often it should run, and health is the age of the last attempt rather than its outcome.

Unauthenticated, deliberately: a check that needs a credential is a check somebody skips configuring.

1{2  "ok": true,3  "checkedAt": "2026-08-17T07:44:12.001Z",4  "jobs": [5    {6      "name": "files-bandwidth",7      "label": "Bandwidth from CloudFront logs",8      "everyMinutes": 60,9      "lastRunAt": "2026-08-17T07:20:03.114Z",10      "lastOk": true,11      "ageMinutes": 24,12      "stale": false13    }14  ]15}

It answers 503 when any job is late or last failed, so a monitor that reads only status codes still notices. It says nothing about what the jobs found — no project names, no counts, no error text.

A job is stale at twice its declared interval. That tolerates one missed firing without crying wolf: a scheduler that skips a run is normal, one that skips two is not. A job that has never run is stale from the start.

Rate limits

Counted per key in Postgres rather than in memory, because this deploys serverless and memory is per-instance — an in-memory limit of 600 is 600 times however many instances happen to be warm, which is not a limit anyone could put a number on.

PLATFORM_RATE_LIMIT_PER_MINUTE changes the ceiling. The customer-facing behaviour is in the Files API.

The auth service has a second limiter underneath this one, and it is not redundant. Ours counts per API key, which bounds what one customer can cost us. Better Auth's counts per end-user IP per endpoint, which is what stops somebody guessing passwords at one of a customer's accounts — the platform limit would happily allow 600 attempts a minute against a single account, because they all present the same valid key.

Its counters go in the project's own database. That is not the default: Better Auth stores them in memory unless told otherwise, and on serverless an in-memory limit of three is three times however many lambdas are warm. The library ships a strict /sign-in/email rule of three attempts per ten seconds, and until the storage was set that rule did nothing at all.

Verified by brute force: eight guesses from one address gave three 401s and then 429s, while a different address signed in normally throughout.

It fails open: if the counter is unreachable, requests are allowed. Refusing would turn a database blip into a total outage for every customer, while allowing costs money for as long as the blip lasts — the second is recoverable and the first is not.

Deleting a project

Deleting is reversible for 30 days, the same window a deleted file gets and for the same reason — it is how long storage keeps the bytes.

The moment it is deleted the project stops resolving keys and its services are told to suspend, so it serves nothing. Nothing is destroyed yet. When the window closes, projects-purge hands the project to each service's deprovision, which is what actually removes the objects, and then the project row and its keys go.

A service that fails to deprovision leaves the project row in place rather than being skipped. A row is a reminder that something is still out there; deleting it would turn a retryable failure into orphaned storage nobody knows about — the job reports it as failed and tries again tomorrow.

This is the only caller of deprovision. Until it existed the method was unreachable — nothing registered the service adapters, so the registry was empty and a deleted project kept its objects forever.

Since the auth service shipped, that call destroys a Postgres database as well as a bucket prefix, which raises the stakes on the failure path. A service that cannot deprovision leaves the project row in place and is retried tomorrow; the Neon project is deleted before the auth service's own row, so a failure leaves a record of what still needs removing rather than a database nothing knows about.

Standing a service up

A project gets its automatic services when it is created, and its on-request ones when somebody asks. That split is a billing decision rather than a technical one: files namespaces a prefix inside a bucket that already exists and costs nothing to stand up, while auth creates a Postgres database and carries a monthly minimum. Handing auth to every project at signup would put a bill on people who never asked for it.

A provisioning failure never fails the project. A customer who has just named a project should not be handed an error about a downstream service; the row records what went wrong and services-resume retries it hourly.

That job does two things in order, and the order matters. It reconciles first — asking each service what state it believes it is in and correcting the platform's copy — because the common case for a failed row is that the work completed and the process died before it could say so. Re-provisioning that is harmless and pointless; correcting it is neither. Only what is still broken afterwards is retried, through resume rather than provision.

provision was unreachable

Until this shipped, nothing anywhere called it. The platform called suspend, resumeService, deprovision and meter — so a newly created project got no service resources at all, and every request to its auth endpoints answered 503 permanently. The one project that worked had been provisioned by hand from a script.

That was the third contract method to be implemented and left unwired, after deprovision and meter, and the shape is always the same: an uncalled method fails no test, because there is nothing to fail. The feature is simply absent.

A test now asserts every method on Service is reachable from outside the services layer. It found two more the moment it was written — resume and status, both of which now have real callers in the sweep above. It is a grep, and a grep only proves a call site exists rather than that it runs at the right moment, which is still the difference between five bugs that shipped and five that could not have.

Provisioning that does real work

The files service's provision records an address and returns — its tenancy is a prefix inside one bucket, so there is nothing to build. Auth gives every project its own Postgres database, so its provision spends money, takes about six seconds, and can fail halfway.

It is idempotent and resumable, and the mechanism is worth knowing: each step's output is its own completion marker, so there is no state column that can disagree with reality. No Neon project id means create one; no sealed connection string means fetch one. Re-running picks up wherever the last attempt stopped.

The step that matters most is the first. Before creating a Neon project it looks for one with the same deterministic name, because a run that died after creating it and before recording the id would otherwise leave a database billing forever with nothing pointing at it. The lookup is filtered on an exact name match rather than trusting Neon's search, which is a substring — adopting the wrong project would point one customer's auth at another customer's users.

Migrating a database per project

A schema change applies N times. There is one source-of-truth Drizzle schema, generated from Better Auth's own table definitions, and one runner that reads the project list and loops:

npm run db:auth:tenant-migrate            # what would happen
npm run db:auth:tenant-migrate -- --yes   # do it

It selects only projects behind the target tag, records each one's version on success, and carries on past a failure rather than aborting — with N databases, some run will die halfway, and the ones already migrated should stay migrated.

Migrations here must be backward-compatible. Not a style rule: a partial rollout always exists mid-flight, and the HTTP driver has no multi-statement transactions, so a migration that fails partway leaves one database between versions. That is recoverable only if the old code still works against the new schema.

Neon's own guide generates a Drizzle config and a CI workflow per project. That is fine for four demo tenants and unworkable at a thousand, where every new customer would be a commit.

Quotas

The platform decides the numbers, the service compares against them. Defaults are 100 GB and a million files per project; either can be overridden per project in the limits column, and on_exceeded chooses between refusing and serving it anyway with a flag in the log.

Checked at upload intent, and only on writes — a read cannot make a project more expensive, so making every list call wait on a quota lookup would be paying for a check that can never fire.

The entitlement is cached for five minutes per process, so lowering a limit takes up to that long to bite and there is no way to clear every serverless instance at once. It is a ceiling, not an emergency stop; the rate limit is the thing that acts immediately.

Orphaned objects

The daily cleanup counts objects that no record points at — the opposite of an abandoned upload, which is a record with no object. It reports rather than deletes: an object with no record looks identical to an object whose record the code cannot see because of a bug, and deleting customer data on the strength of a query returning nothing is not a trade worth making. The count lands in the job result.

What is not here yet

Said plainly because a gap you know about is cheaper than one you discover:

• Retries. A failed job waits for its next scheduled run.
• Anything watching the request path. Health covers scheduled work only.