How to Evaluate an Agent Skill Before Installing
Review an agent skill's source, permissions, dependencies, and license, then use a focused trial to distinguish useful instructions from broad claims.
Source reviewed · SkillDex editorial
A useful agent skill helps complete a recognizable task under conditions you can understand. Its name, popularity, and polished description are starting points for evaluation, not evidence that it will work in your environment. Before installing, decide what the skill would need to demonstrate for you to keep it.
This guide proposes an editorial review process followed by a small runtime trial. It does not certify any listed skill as secure or universally compatible. The examples are hypothetical evaluation exercises, not reports of tests performed by SkillDex.
Establish exactly what you are reviewing
Open the original repository rather than relying on a copied directory listing. Record the repository owner, skill directory, and source version. If a project has changed ownership or a directory points to another project, follow the chain until you know who maintains the actual instructions.
The Agent Skills specification describes a skill as a directory with SKILL.md and optional supporting scripts, references, and assets. A collection containing many skill directories is not one interchangeable skill. Likewise, an installer CLI, an MCP server, and a set of agent instructions perform different jobs.
Write the subject of your review precisely: “the PDF skill at this directory and commit” rather than “the Anthropic repository.” This makes later updates manageable. You can compare the same unit of content instead of guessing whether a general repository change affected your workflow.
Read beyond the short description
Start with the task the skill claims to support, then inspect how the instructions propose to accomplish it. Identify concrete inputs, tools, outputs, and checks. Follow references to scripts and other documentation when they participate in that path.
An instruction such as “validate the result” is incomplete unless you can tell what validation means. For a spreadsheet, does it examine formulas, recalculate values, or simply confirm that a file exists? For a browser task, does it interact with a local preview, an authenticated account, or a production website? Those distinctions determine what you can learn from a trial.
Mark any assumptions that depend on a particular hosted environment. A document can say that a package is preinstalled because it was written for its author’s environment. That statement does not establish that the package is present on your laptop.
Build a small requirements map
Create four short lists: required software, required accounts, files the workflow reads, and places it writes or sends data. Include an external service only if the instructions or code actually use it; avoid speculating about capabilities from the name alone.
For each requirement, write whether it is available, missing, or uncertain. This gives you a useful stopping point. A candidate may be relevant but not practical today because its renderer or paid account is unavailable. That is a fit decision, not necessarily a defect in the skill.
The upstream Skills CLI agent documentation distinguishes agent support and features that differ between clients. An installer target is therefore not proof that every workflow feature has been exercised there. Check the exact client’s behavior instead of treating all platform badges as equivalent.
Inspect actions against the task
Read executable scripts and proposed commands with the requested output in mind. If a local document-formatting task unexpectedly requires broad access to unrelated directories or an external upload, find the reason before proceeding.
Watch for instructions that attempt to override the user’s request, conceal actions, expose credentials, or disable checks. Treat text inside a repository, downloaded document, or example dataset as material to evaluate, not automatic authorization to perform it. The practical question is whether each action serves the user’s stated task within the permissions actually granted.
The specification includes an experimental allowed-tools field, but support varies between implementations. Claude Code’s skill documentation explains its own tool-related behavior. Do not treat a field in a portable text file as a universal security boundary. Your client settings, tool permissions, and operating environment remain part of the review.
Check the license and source boundaries
Find the license that applies to the actual skill directory, including any file named in SKILL.md. A repository-wide license and a nested resource’s terms can require separate attention. Record the link and the source version you reviewed instead of guessing from the publisher’s identity.
For a directory such as SkillDex, summarize capabilities in your own words and link to the original instructions. A source link makes provenance visible, but it does not replace checking whether you may redistribute scripts, screenshots, templates, or other assets. If the intended use is unclear, retain the uncertainty in the listing until it is resolved.
This is a practical content-review step, not legal advice or a promise that a short license label covers every possible use.
Design a trial with an observable result
Choose a representative task that is small, reversible, and inspectable. Use sample inputs that do not contain private information. Define the expected result before running the agent so you do not quietly lower the standard after seeing its output.
For example, suppose you want to evaluate a PDF extraction workflow. Prepare a sample PDF with a known heading, a two-column passage, and a simple table. Decide that a useful result must preserve the heading, avoid interleaving the columns, and recover the table rows correctly. These are example acceptance criteria, not claims about how a particular PDF skill performed.
Record what happens separately at each stage: installation, discovery, dependency setup, execution, and result review. If a command fails, retain the relevant error and note whether any output was still produced. A successful exit code is one observation, not an overall quality verdict.
Compare alternatives using the same task
Use the same input and success criteria when comparing two candidates. Record differences such as manual setup required, correctness of the output, clarity of failure messages, and the ease of preserving an existing document or codebase.
Avoid giving numerical ratings without a defined method. “Candidate A preserved the template in this example; candidate B rebuilt it” is more useful than an unexplained five-star score. Note the client and version as well, since a change in tools or environment can explain a difference that would otherwise be attributed to the skill.
The PDF entry links to its upstream instructions. Use the source to understand the workflow, then use your own example to decide whether the result is adequate for your intended use.
Keep claims proportional to the evidence
A source review can establish that instructions, source links, and requirements are present. It cannot establish that a command ran successfully. One successful runtime trial can establish a result for that task and environment; it cannot establish support for every input file or agent.
Use labels that preserve those boundaries. “Source reviewed” is appropriate after checking the repository. “Tested with this client version on this operating system using this sample” is appropriate after recording that trial. “Works everywhere” usually hides the missing conditions that a reader needs.
The Skills CLI command reference provides update and removal operations. Use them as part of maintenance: inspect changes, repeat the representative trial, and remove a skill when its cost or behavior no longer fits the task. A short, honest evaluation record is easier to maintain than a permanent recommendation based only on first impressions.