> Scheduled jobs, how a failure surfaces, and the one endpoint a monitor should watch. # Operating it Most of this platform runs on request. What does not is a handful of scheduled jobs, and those are the part that can stop working without anybody noticing — which is what this page is about. ## The scheduled jobs | Job | Runs | Does | |---|---|---| | `services-resume` | hourly, at :35 | Finishes service provisioning that failed or never came back | | `files-bandwidth` | hourly, at :20 | Reads CloudFront's access logs into per-project bandwidth | | `files-cleanup` | 03:10 daily | Sweeps uploads that were intended and never confirmed | | `files-storage` | 03:40 daily | Samples what each project occupies in storage | | `auth-cleanup` | 04:00 daily | Sweeps the console's own sign-in throttle | | `projects-purge` | 04:20 daily | Destroys projects past their restore window | | `auth-users` | 04:40 daily | Counts each project's end users, one database at a time | | `billing-charge` | 05:00 daily | Prices yesterday's usage and charges it against credit | The ordering is the load-bearing part of that table, and every gap in it is a decision. **Bandwidth runs at twenty past, not on the hour.** CloudFront delivers access logs with a lag, and starting on the hour races the delivery. **Cleanup runs before the storage sample, not after.** Swept objects have to be gone *before* the thing that measures them, or a customer is billed for uploads that were abandoned and deleted the same night. **`auth-users` runs after the purge and before the charge.** After, so a project destroyed last night is not counted and billed on its way out. Before, so the number being charged was taken today rather than yesterday. **`auth-cleanup` is not the auth service.** It sweeps the *console's* sign-in throttle and always has. The name predates there being an auth service and now reads as though it belonged to one; renaming it would change its cron path and orphan its run history, so the label carries the correction instead. ## When a job fails Every job runs through a wrapper that records the attempt, its duration and its outcome, then mails `PLATFORM_OWNER_EMAILS`. ```json A failed run's response { "ok": false, "job": "files-storage", "error": "The job failed. The detail was recorded." } ``` The response carries no detail on purpose — a job's error can contain a connection string, and this route is one misconfiguration away from being public. The detail is in the record and in the mail. ## When a job doesn't run at all This is the failure that alerting cannot catch. A job that never fires raises no error, sends no mail, and looks exactly like a quiet night — which is also the state after deploying a schedule that does not work. So every job declares how often it should run, and health is the *age* of the last attempt rather than its outcome. ## `GET /api/health` Unauthenticated, deliberately: a check that needs a credential is a check somebody skips configuring. ```json Healthy { "ok": true, "checkedAt": "2026-08-17T07:44:12.001Z", "jobs": [ { "name": "files-bandwidth", "label": "Bandwidth from CloudFront logs", "everyMinutes": 60, "lastRunAt": "2026-08-17T07:20:03.114Z", "lastOk": true, "ageMinutes": 24, "stale": false } ] } ``` It answers **503** when any job is late or last failed, so a monitor that reads only status codes still notices. It says nothing about what the jobs found — no project names, no counts, no error text. A job is **stale** at twice its declared interval. That tolerates one missed firing without crying wolf: a scheduler that skips a run is normal, one that skips two is not. A job that has never run is stale from the start. ## Rate limits Counted per key in Postgres rather than in memory, because this deploys serverless and memory is per-instance — an in-memory limit of 600 is 600 times however many instances happen to be warm, which is not a limit anyone could put a number on. `PLATFORM_RATE_LIMIT_PER_MINUTE` changes the ceiling. The customer-facing behaviour is in the [Files API](/docs/files-api). **The auth service has a second limiter underneath this one**, and it is not redundant. Ours counts per API key, which bounds what one customer can cost us. Better Auth's counts per end-user IP per endpoint, which is what stops somebody guessing passwords at one of a customer's accounts — the platform limit would happily allow 600 attempts a minute against a single account, because they all present the same valid key. Its counters go in **the project's own database**. That is not the default: Better Auth stores them in memory unless told otherwise, and on serverless an in-memory limit of three is three times however many lambdas are warm. The library ships a strict `/sign-in/email` rule of three attempts per ten seconds, and until the storage was set that rule did nothing at all. Verified by brute force: eight guesses from one address gave three 401s and then 429s, while a different address signed in normally throughout. It **fails open**: if the counter is unreachable, requests are allowed. Refusing would turn a database blip into a total outage for every customer, while allowing costs money for as long as the blip lasts — the second is recoverable and the first is not. ## Deleting a project Deleting is reversible for 30 days, the same window a deleted file gets and for the same reason — it is how long storage keeps the bytes. The moment it is deleted the project stops resolving keys and its services are told to suspend, so it serves nothing. Nothing is destroyed yet. When the window closes, `projects-purge` hands the project to each service's `deprovision`, which is what actually removes the objects, and then the project row and its keys go. A service that fails to deprovision leaves the project row in place rather than being skipped. A row is a reminder that something is still out there; deleting it would turn a retryable failure into orphaned storage nobody knows about — the job reports it as `failed` and tries again tomorrow. **This is the only caller of `deprovision`.** Until it existed the method was unreachable — nothing registered the service adapters, so the registry was empty and a deleted project kept its objects forever. Since the auth service shipped, that call destroys a Postgres database as well as a bucket prefix, which raises the stakes on the failure path. A service that cannot deprovision leaves the project row in place and is retried tomorrow; the Neon project is deleted *before* the auth service's own row, so a failure leaves a record of what still needs removing rather than a database nothing knows about. ## Standing a service up A project gets its `automatic` services when it is created, and its `on-request` ones when somebody asks. That split is a billing decision rather than a technical one: files namespaces a prefix inside a bucket that already exists and costs nothing to stand up, while auth creates a Postgres database and carries a monthly minimum. Handing auth to every project at signup would put a bill on people who never asked for it. **A provisioning failure never fails the project.** A customer who has just named a project should not be handed an error about a downstream service; the row records what went wrong and `services-resume` retries it hourly. That job does two things in order, and the order matters. It **reconciles** first — asking each service what state it believes it is in and correcting the platform's copy — because the common case for a `failed` row is that the work completed and the process died before it could say so. Re-provisioning that is harmless and pointless; correcting it is neither. Only what is still broken afterwards is retried, through `resume` rather than `provision`. ### `provision` was unreachable Until this shipped, nothing anywhere called it. The platform called `suspend`, `resumeService`, `deprovision` and `meter` — so a newly created project got no service resources at all, and every request to its auth endpoints answered 503 permanently. The one project that worked had been provisioned by hand from a script. That was the third contract method to be implemented and left unwired, after `deprovision` and `meter`, and the shape is always the same: an uncalled method fails no test, because there is nothing to fail. The feature is simply absent. A test now asserts every method on `Service` is reachable from outside the services layer. It found two more the moment it was written — `resume` and `status`, both of which now have real callers in the sweep above. It is a grep, and a grep only proves a call site exists rather than that it runs at the right moment, which is still the difference between five bugs that shipped and five that could not have. ## Provisioning that does real work The files service's `provision` records an address and returns — its tenancy is a prefix inside one bucket, so there is nothing to build. Auth gives every project its own Postgres database, so its `provision` spends money, takes about six seconds, and can fail halfway. It is idempotent and resumable, and the mechanism is worth knowing: each step's *output* is its own completion marker, so there is no state column that can disagree with reality. No Neon project id means create one; no sealed connection string means fetch one. Re-running picks up wherever the last attempt stopped. The step that matters most is the first. Before creating a Neon project it looks for one with the same deterministic name, because a run that died after creating it and before recording the id would otherwise leave a database billing forever with nothing pointing at it. The lookup is filtered on an exact name match rather than trusting Neon's `search`, which is a substring — adopting the wrong project would point one customer's auth at another customer's users. ## Migrating a database per project A schema change applies N times. There is one source-of-truth Drizzle schema, generated from Better Auth's own table definitions, and one runner that reads the project list and loops: ```bash npm run db:auth:tenant-migrate # what would happen npm run db:auth:tenant-migrate -- --yes # do it ``` It selects only projects behind the target tag, records each one's version on success, and carries on past a failure rather than aborting — with N databases, some run will die halfway, and the ones already migrated should stay migrated. **Migrations here must be backward-compatible.** Not a style rule: a partial rollout always exists mid-flight, and the HTTP driver has no multi-statement transactions, so a migration that fails partway leaves one database between versions. That is recoverable only if the old code still works against the new schema. Neon's own guide generates a Drizzle config and a CI workflow per project. That is fine for four demo tenants and unworkable at a thousand, where every new customer would be a commit. ## Quotas The platform decides the numbers, the service compares against them. Defaults are 100 GB and a million files per project; either can be overridden per project in the `limits` column, and `on_exceeded` chooses between refusing and serving it anyway with a flag in the log. Checked at upload intent, and only on writes — a read cannot make a project more expensive, so making every list call wait on a quota lookup would be paying for a check that can never fire. **The entitlement is cached for five minutes per process**, so lowering a limit takes up to that long to bite and there is no way to clear every serverless instance at once. It is a ceiling, not an emergency stop; the rate limit is the thing that acts immediately. ## Orphaned objects The daily cleanup counts objects that no record points at — the opposite of an abandoned upload, which is a record with no object. It **reports** rather than deletes: an object with no record looks identical to an object whose record the code cannot see because of a bug, and deleting customer data on the strength of a query returning nothing is not a trade worth making. The count lands in the job result. ## What is not here yet Said plainly because a gap you know about is cheaper than one you discover: - **Retries.** A failed job waits for its next scheduled run. - **Anything watching the request path.** Health covers scheduled work only.