Critico

relai.critico.Critico(client)

Critico orchestrates evaluation of an AI agent using a configurable set of evaluators.

Parameters:

Name	Type	Description	Default
client	AsyncRELAI	An instance of the AsyncRELAI client to interact with the RELAI platform.	required

add_evaluators(evaluators, evaluator_group='default')

Adds a new evaluator to Critico to a specific evaluator group.

Parameters:

Name	Type	Description	Default
evaluator_group	str	The name of the evaluator group.	'default'
evaluators	dict[Evaluator, float]	A dictionary where keys are Evaluator objects and values are their corresponding weights for this evaluator group. If None, no evaluators are associated with this evaluator group initially. Defaults to None.	required

evaluate(agent_logs) async

Evaluates a list of AI agent logs against their corresponding evaluator groups using the configured evaluators.

For each AgentLog, Critico identifies the associated evaluator group and runs all evaluators configured for that group. It then aggregates the individual evaluator results (scores and feedback) into a single CriticoFeedback object.

Parameters:

Name	Type	Description	Default
agent_logs	list[AgentLog]	A list of AgentLog objects to be evaluated. Each log must contain a simulation_tape.evaluator_group that corresponds to an evaluator group added to Critico.	required

Returns:

Type	Description
list[CriticoLog]	list[CriticoFeedback]: A list of CriticoFeedback objects, where each object summarizes the evaluation outcome for one AgentLog.

Raises:

Type	Description
KeyError	If a evaluator_group within an AgentLog is not found among Critico's managed evaluator groups.

report(critico_logs) async

Submits the critico logs (as multiple CriticoLog objects) to the RELAI platform.

Parameters:

Name	Type	Description	Default
critico_logs	list[CriticoLog]	A list of CriticoLog objects containing the evaluation results for each AgentLog.	required

Raises:

Type	Description
RELAIError	If any CriticoLog does not have a valid trace_id.

reweight_evaluator(evaluator_group, evaluator, weight)

Adjusts the weight of a specific evaluator in a given evaluator group.

Evaluator weights influence their contribution to the aggregate score during the evaluate() operation.

Parameters:

Name	Type	Description	Default
evaluator_group	str	The name of the evaluator group.	required
evaluator	Evaluator	The Evaluator object whose weight is to be adjusted.	required
weight	float	The new weight (a positive float) for the evaluator. Higher weights give more prominence to its results in aggregation.	required

Raises:

Type	Description
RELAIError	If the evaluator_group is not found in Critico, or if the evaluator is not associated with the specified evaluator_group.
ValueError	If the specified weight is not a positive float.