Promptfoo

We put Promptfoo in Assess in 2023 for testing prompts. We still use it for evals, including agent tool calls and CI gates, but now also for red teaming. We recommend trialing it for both. To red-team an app, you describe what it's for, and Promptfoo generates attacks for the risks you choose, applies attack strategies and grades the responses.

We used Promptfoo to red-team a production conversational assistant with domain-specific safety rules. It turned ad hoc manual probing into repeatable tests rerun after every fix. It also works with stateful guardrails: the guardrails are the runtime control, and Promptfoo tests that they hold. Promptfoo is now owned by OpenAI, although it remains MIT-licensed and is model agnostic. Some of Promptfoo's more advanced simulated attacks rely on its hosted service, so make sure you're comfortable with what data leaves your organization.

promptfoo enables test-driven prompt engineering. While integrating LLMs in applications, tuning of the prompts to produce optimal, consistent outputs can be time-consuming. You can use promptfoo both as a CLI and a library to systematically test prompts against predefined test cases. The test case, along with assertions, can be set up in a simple YAML config file. This config includes the prompts being tested, the model provider, the assertions and the variable values that will be substituted in the prompts. promptfoo supports many assertions, including checking for equality, JSON structure, similarity, custom functions or even using an LLM to grade the model outputs. If you're looking to automate feedback on prompt and model quality, do assess promptfoo.