Fewer brittle tests and more portable skills
I worked on ChatPRD’s Google Drive integration and CI, and reviewed the How I AI skill library for instructions that depended too much on a particular AI client. Two useful audits came out of the day: what our tests actually protected, and what our skills actually needed to run. The Google Docs guides cover reviews and linked exports.
Audit test value with Codex without dropping important regressions
I asked Codex to make ChatPRD’s CI faster, then asked a more basic question: did we need all of those tests? I specifically wanted to cut assertions about exact prompts and other details that could change without breaking the product.
Codex inspected the tests and recorded what each proposed removal was protecting. We removed nine low-value or redundant tests and trimmed exact wording, editorial-heading, CSS-class, and duplicate rendering assertions. The audit kept the important checks around permissions, privacy, persistence, publication, and recovery that prevents repeated paid calls.
The remaining suite passed locally with the existing opt-in skips unchanged. The useful deliverable was the audit ledger as much as the smaller suite: it explained each removal and pointed to the behavior still covered elsewhere. I could review what protection remained instead of judging the change by a test count.
I would use the same request again when an AI-written test suite starts to freeze implementation details. Ask what failure each test catches, then make a small, reviewable change. A passing build alone does not replace tests for application behavior.
# Audit test value before deleting tests
Inspect the test suite and its recent failures. Identify tests that primarily assert exact prompt wording, cosmetic details, internal structure, or behavior already covered elsewhere.
For each candidate, explain the actual failure it catches and whether removing it would leave a gap. If the behavior matters, replace the brittle assertion with a behavior check rather than deleting the protection.
Preserve authorization, privacy, persistence, recovery, publication, concurrency, and safeguards against duplicate paid operations. Preserve existing opt-in boundaries; do not invoke paid services to run this audit.
Make a small change and keep an audit ledger with the removed assertion, rationale, and surviving coverage. Run the relevant tests and required repository checks. Report skips and distinguish local results from hosted CI. Do not change branch protections or merge the change.Audit AI skills for unnecessary client dependencies with Codex
Once the How I AI skills were available through the plugin, I worried that too many instructions assumed the reader was using Codex or Claude Code. I had Codex audit the published skill text, then asked for parallel reviewers to revise the unnecessary dependencies.
We needed to separate three things: which tool the guest used, which capabilities the task required, and which client the reader happened to be using. A source attribution should stay intact. A task that genuinely depends on a platform feature should keep that requirement. An instruction to open a particular coding client just to write a plan usually needs a different treatment.
The review covered 286 skills. We published revisions to 236 and left 50 unchanged. The revised instructions used the active client’s available tools and described useful handoffs when it could not perform a required step. Codex read the changes back from the CMS and checked representative downloads.
This was an editorial portability pass, not execution testing of all the workflows. It made the instructions less likely to send someone to a different app unnecessarily while keeping the actual tools and source episodes visible.
# Audit a skill for unnecessary client dependencies
Read the full skill and its source attribution. Classify each platform reference as source attribution, an example, a genuine task requirement, or an unnecessary assumption about the client executing the instructions.
Preserve attribution and genuine requirements. Rewrite accidental client assumptions in terms of the capability needed: source access, an independent review context, file editing, browser interaction, or execution. Avoid blanket name replacement.
Use the current client’s available tools where they satisfy the requirement. If a capability is missing, describe the useful work it can still complete and the concrete handoff required to finish. Never claim it executed a step it could only describe.
Preserve the intended outcome, examples, safeguards, and source links. Return a concise change rationale with the revised skill. Keep edits in draft for review. After an approved save, read the content back and check that any downloadable version matches. Report editorial review separately from execution testing.