MANTRA (MANual-to-Test-tRAnslation) is a pipeline that automatically turns a natural-language procedural manual and a tool schema into machine-checkable compliance benchmarks for tool-using LLM agents. It evaluates compliance over an agent's trace of tool calls, not just its final answer, and grades tasks deterministically. Correctness of each generated task is established by cross-validating two independently produced formal objects with an SMT solver.

The poster is interactive, click on the boxes to learn more about them.