AgentEval

Overview

AgentEval is an abstract class designed for evaluating the performance of Language Learning Models (LLMs) in the Automata library. It operates by generating evaluation results for a specified set of instructions and expected actions. “Evaluation” here includes processing the results of a session, and comparing these results against an expected sequence of actions to evaluate how closely the model’s actions followed the expected sequence. Inheritances of this class should implement the generate_eval_result and process_result methods.

Interface Methods

generate_eval_result(self, exec_input: AutomataTask, expected_output: List[Action], executor: AutomataTaskExecutor, *args, **kwargs) -> EvalResult

This method is used to generate an evaluation result for a given set of instructions (exec_input) and expected actions (expected_output). The executor parameter is an instance of AutomataTaskExecutor that is used to execute the task.

process_result(self, expected_actions: List[Action], process_input: Sequence[LLMChatMessage], *args, **kwargs) -> EvalResult

This method processes the result of an evaluation. It takes in an expected list of actions and a sequence of LLMChatMessage instances to process the evaluation.

Usage Example

from automata.eval.agent.agent_eval import AgentEval
from automata.eval.agent.agent_eval_result import AgentEvalResult
from automata.tasks.task_executor import AutomataTaskExecutor
from typing import List
from automata.common.types import Action, AutomataTask

class MyAgentEval(AgentEval):

    def generate_eval_result(self, exec_input: AutomataTask, expected_output: List[Action], executor: AutomataTaskExecutor) -> AgentEvalResult:
        # you need to implement this method based on how you want to evaluate.
        pass

    def process_result(self, expected_actions: List[Action], process_input: Sequence[LLMChatMessage]) -> EvalResult:
        # you need to implement this method based on how you want to process the evaluation.
        pass

# Create an instance of MyAgentEval
my_agent_eval = MyAgentEval()

Limitations

The primary limitations associated with AgentEval are the need for each inheritor to implement its own versions of generate_eval_result and process_result methods. This requires a clear understanding of the specific evaluation process required for each unique learning model. This evaluation process must also be implementable in a manner compatible with AgentEval’s methods.

Follow-up Questions:

How can we generalize AgentEval evaluation methods to be applicable to a wider range of learning models?
Can we simplify AgentEval interfaces while maintaining their function for evaluations?