Require evidence of discovery coverage
Ask which sources the software reads and which assets it cannot see. Test hosted AI services, agent runtimes, service identities and locally configured tools with synthetic data. A connector list does not prove it reconciles every identity or recognises the same system across several sources.
Manual and automated work
Manual records can capture purpose and accountable ownership well when teams maintain them. Automated discovery can help find configured assets and changes, but may not know intended use or approval status. Require an owner validation workflow for either approach and explicit fields for unresolved context.
Integrations
Ask vendors to demonstrate each source and destination that matters in your environment. Record the product, connector or import method, permissions requested, fields collected, sync schedule, error handling, and which records need owner validation. A connection to an identity, cloud or development platform does not prove that every AI system, model, agent, MCP server, or locally configured tool is visible. Keep a written list of sources that were tested and the gaps that remain.
Compare answers in a simple evidence matrix. These are buyer-defined criteria, not a vendor score or tested product ranking:
| Question | Evidence to request | Example status |
|---|---|---|
| Can it import our deployment and identity records? | Demonstrate a read-only connection or sample import; list required permissions and unavailable fields. | Partial: two sample sources shown; remaining sources untested. |
| Can owners validate purpose and permitted actions? | Show the owner workflow, review history, and how unresolved fields are represented. | Demonstrated in the synthetic trial; production workflow not yet reviewed. |
| Can we export records and evidence references? | Inspect a sample export, stable IDs, timestamps, and links to source evidence. | Unproven until the buyer checks a sample export. |
Use the NIST AI RMF Map Playbook as voluntary context for documenting intended purpose and deployment context, not as a product certification or required integration list. Criteria and example statuses above are illustrative and have not been tested against vendors.
Compare the evidence you collect with the manual inventory builder, AI inventory template, and AI agent inventory template. The builder is browser-side and manual; it flags missing context, downloads CSV and uses browser print/save-to-PDF, without asset discovery, integrations, registry API, account access or completeness verification.
Evaluation criteria
Assess stable identifiers, evidence provenance, change history, owner assignments, tool/permission relationships, review workflow and export portability. Mark each demonstrated, partial or unproven. Keep legal classification separate from vendor risk labels, and avoid unsupported rankings of products you have not tested.
Worked proof of concept
Load the three-agent synthetic example, then remove an owner and add a write permission. Ask the vendor to show the gap, preserve the previous state and route review. Verify that exports retain evidence references and that retiring an agent does not erase its historical review trail.
Costs and implementation questions
Use the vendor’s real quote and determine whether it counts agents, identities, connectors or seats. Ask about onboarding effort, maintenance, data egress and read-only access options. No prices are asserted here. The free local builder can help define the fields your trial must preserve before choosing software.
Run a repeatable evaluation
Before a trial, define which asset types, business units and environments are in scope. Agree on a small synthetic dataset and a stable identifier for each sample. Ask the vendor to show how it handles duplicates, an unknown owner, a changed permission and a retired deployment. Record what was demonstrated, what was described but not shown, and what was not tested. These are buyer-selected criteria, not a tested comparison of named products.
Separate discovery evidence from governance evidence. A connector may return an account, endpoint, model name or service identity, while an owner must still confirm purpose, users, data and decision impact. Ask how the product preserves source and collection time, how stale records are surfaced, and whether a human can correct an incorrect match. Verify export portability and retention using a sample file before procurement.
For costs, request a quote with the assumed number of users, identities, connectors, environments, data volume and support level. Ask which onboarding and maintenance tasks your team must perform and whether pricing changes when assets are inactive or duplicated. Compare the written quote and implementation effort; do not infer pricing from a feature page or another buyer's estimate. No vendor prices or performance scores are asserted in this guide.
The local builder can help agree on a starter schema: agent, owner, purpose, platform, environment, tools, data class, permissions, autonomy and review date. Its triage and CSV export are manual and do not prove that a purchased product found every asset. Review the AI inventory template and AI agent inventory template before writing acceptance criteria.