AI Tools

AI Tools Maintenance Checklist

Subscription and Seat Audit

Subscriptions accumulate. People change roles, teams reorganize, and a tool bought for a finished project keeps billing. Once a quarter, list every active plan with its renewal date, seat count, and the last time each named user signed in or produced work. Reclaim dormant seats, and look for overlapping capability — two assistants doing the same drafting job is usually one subscription too many.

Read the tier before renewing, not after. Vendors move features between plans, so a team that outgrew its message allowance may be paying overage charges a higher tier would absorb. Check whether the plan bills per seat or per unit of consumption, because the right answer changes as headcount and volume drift apart.

Key Rotation and Permission Hygiene

Inventory every key and token that reaches a model endpoint, then confirm each one is still needed and still scoped to the minimum it does. Keys created for a prototype, shared in a group chat, or scoped to an entire organization are the first to retire. Rotate on a fixed cadence — more often for credentials that touch production data — and revoke immediately when a person leaves or a device is lost.

Set an expiry where the platform supports one, and prefer short-lived tokens from a service account over static keys. Confirm permissions still match the job: read-only where the task only reads, single workspace rather than organization-wide. A credential that outlives its purpose is a standing invitation.

Usage, Quotas, and Rate Limits

Model spend is easy to lose track of because it scales with traffic and with prompt size. Watch three separate ceilings: the monthly allowance on your plan, the per-minute request or token rate limit, and your own budget. A retry loop, a stuck automation, or a prompt that quietly grew to include an entire document can consume a quota in an afternoon while the interface still behaves normally.

Set spend alerts below the plan ceiling, log which workflow or seat accounts for the most tokens, and trim the expensive paths — cache repeated context, send the retrieved passages instead of the whole corpus, and use a smaller model where the task does not need the largest one. When a rate limit bites during peak hours, choose deliberately between queueing requests, batching them, or paying for a higher tier.

Re-testing After Model Version Changes

Treat a model version bump as a change to your software, because that is what it is. Outputs can shift in wording, length, formatting, refusal behaviour, and how strictly a system prompt is obeyed. Template slots, validation rules, and response parsers written against the previous version break first.

Keep a small regression set of representative inputs with the outputs you expect, run it whenever a version changes or a deprecation notice arrives, and pin versions where the vendor allows it so the change is your decision rather than a surprise. Store the version alongside each retained output, so a later quality question can be traced to the model that produced it.

Versioning Prompts and Evaluation Sets

Prompts are production assets and deserve the same treatment as code: a repository, a change history, and a note explaining why each instruction exists. Store the system prompt, its examples, and the expected output format together with the model version they were tuned against. A prompt library that lives only inside a vendor’s interface disappears when the subscription does; a file in your own repository survives a migration.

Keep the evaluation set beside it. Golden inputs with known-good answers, plus a scoring rubric that states what counts as a pass, turn subjective arguments into a repeatable check. When someone proposes rewording, they run the set and show the result instead of asserting that the new phrasing reads better.

Retention, Logging, and Privacy Settings

Administrative settings drift. A workspace owner changes a default, a new integration is connected under different terms, or a vendor revises its policy language, and the data-handling posture you approved no longer matches what is running. Once or twice a year, confirm the retention window for prompts and outputs, check that training opt-outs are still enabled on every workspace, and verify which regions process requests.

Audit the log side as well: conversation history visible to admins, exports the team has downloaded, and connected apps holding access to stored content. Keep the data processing agreement on file, review the subprocessor list, and make sure the people entering sensitive material know which tool is sanctioned. Settings nobody re-reads are settings nobody maintains.

Retiring Unused Integrations

Every integration is a live door. When a workflow is retired, its key, OAuth grant, webhook subscription, and scheduled job often survive — still authenticated, still able to write. Walk the list of connected applications and automations, match each against a current owner and a current business need, and remove whatever nobody claims.

Removal means more than disconnecting one side. Revoke the token at the identity provider, delete the webhook endpoint so the vendor stops delivering events, and disable the scheduled job. Then decide what happens to the data the integration produced: archive it in an export or schedule it for deletion under your retention policy. A dormant connector left in place is an unmonitored path back into your systems.

Periodic Output Quality Checks

Capability decays quietly. Prompts accumulate extra context, source documents change, a model version shifts underneath, and an output that was accurate when it shipped can drift while still looking correct. Sample real production output on a schedule — the longest inputs, the edge cases, the drafts a reviewer flagged — and score them against the rubric stored with your evaluation set.

Track how often each workflow’s output is edited before it is used. A rising correction rate is the earliest visible symptom that something upstream moved: an altered template, a stale knowledge source, or a model that now formats answers differently. Feed what you find back into the prompt, the retrieval corpus, or the decision to keep that workflow automated at all.

FAQ

How often should we review AI subscriptions?

A quarterly review catches most drift, because seat counts otherwise lag behind staffing changes by weeks and renewal dates arrive whether you are ready or not. Schedule it before the renewal date and check usage per seat rather than assuming everyone on the plan still signs in.

Do we have to re-test everything when a model updates?

You need to re-test the workflows that parse output or feed consequential decisions; a small regression set of representative inputs is enough to surface format and behaviour drift. The rest of the tool’s uses can get a spot check rather than a full pass.

What is the risk of leaving old API keys active?

An unused key is still a credential — it works from anywhere it leaks to, and it usually carries broader permissions than the work it was created for. Rotate and revoke on a schedule, and treat any key that has appeared in a shared document or a prompt as already exposed.

Should prompts live in the vendor’s tool or our own repository?

Your own repository. Keeping prompts, examples, and evaluation sets under version control lets you review changes, compare revisions, and move providers without reconstructing the work from chat history. Copy them into the vendor interface as a deployment target, not as the source of truth.

What should a maintenance review actually produce?

A short list of decisions: seats to reclaim, keys to rotate, integrations to retire, and prompts to re-run against the evaluation set. If a review produces no decisions at all, either it happened too recently or nobody is reading the usage reports.