Skip to content

Commit ef5b104

Browse files
penguine-ippenguine-ip
authored
Merge pull request confident-ai#2565 from seankelley-dt/2564-fix-expected-tools-csv-parsing
Fix: Parse CSV Single-Turn Golden ToolCalls as JSON objects
2 parents a82f1a2 + 0a735f4 commit ef5b104

2 files changed

Lines changed: 57 additions & 36 deletions

File tree

‎deepeval/dataset/dataset.py‎

Lines changed: 41 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -531,26 +531,47 @@ def get_column_data(df: pd.DataFrame, col_name: str, default=None):
531531
df, retrieval_context_col_name, default=""
532532
)
533533
]
534-
tools_called = [
535-
(
536-
tool_called.split(tools_called_col_delimiter)
537-
if tool_called
538-
else []
539-
)
540-
for tool_called in get_column_data(
541-
df, tools_called_col_name, default=""
542-
)
543-
]
544-
expected_tools = [
545-
(
546-
expected_tool.split(expected_tools_col_delimiter)
547-
if expected_tool
548-
else []
549-
)
550-
for expected_tool in get_column_data(
551-
df, expected_tools_col_name, default=""
552-
)
553-
]
534+
535+
tools_called = []
536+
for tools_called_str in get_column_data(
537+
df, tools_called_col_name, default=""
538+
):
539+
if tools_called_str:
540+
try:
541+
# Try loading JSON-serialized ToolCall objects
542+
parsed_tools = [
543+
ToolCall(**tool)
544+
for tool in trimAndLoadJson(tools_called_str)
545+
]
546+
tools_called.append(parsed_tools)
547+
except ValueError or json.JSONDecodeError:
548+
# Fallback to simple split on delimiter
549+
tools_called.append(
550+
tools_called_str.split(tools_called_col_delimiter)
551+
)
552+
else:
553+
tools_called.append([])
554+
555+
expected_tools = []
556+
for expected_tools_str in get_column_data(
557+
df, expected_tools_col_name, default=""
558+
):
559+
if expected_tools_str:
560+
try:
561+
# Try loading JSON-serialized ToolCall objects
562+
parsed_tools = [
563+
ToolCall(**tool)
564+
for tool in trimAndLoadJson(expected_tools_str)
565+
]
566+
expected_tools.append(parsed_tools)
567+
except ValueError or json.JSONDecodeError:
568+
# Fallback to simple split on delimiter
569+
expected_tools.append(
570+
expected_tools_str.split(expected_tools_col_delimiter)
571+
)
572+
else:
573+
expected_tools.append([])
574+
554575
comments = get_column_data(df, comments_key_name)
555576
name = get_column_data(df, name_key_name)
556577
source_files = get_column_data(df, source_file_col_name)

‎tests/test_core/test_datasets/goldens.csv‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
1-
input,actual_output,expected_output,retrieval_context,context,name,comments,source_file
2-
What is the tallest mountain in the world?,Mount Everest,Mount Everest,Mountain heights list|Moutain list,Geography|Mountains,Name 1,Comments 1,
3-
Solve 5 * 7,35,35,Basic arithmetic,Math|Multiplication,Name 2,Comments 2,
4-
Define 'photosynthesis',Process by which green plants convert light energy into chemical energy,Process by which green plants convert light energy into chemical energy,Biology textbooks|Science API,Biology|Photosynthesis,Name 3,Comments 3,
5-
What is the capital of France?,Paris,Paris,World capitals list,Geography|Capital Cities,Name 4,Comments 4,
6-
What is the square root of 64?,8,8,Basic arithmetic,Math|Square Roots,Name 5,Comments 5,
7-
What causes rain?,Rain is caused by the condensation of water vapor in the atmosphere.,Rain is caused by the condensation of water vapor in the atmosphere.,Weather patterns data,Science|Meteorology,Name 6,Comments 6,
8-
Translate 'Hello' into Spanish.,Hola,Hola,Translation data,Language|Translation,Name 7,Comments 7,
9-
What is the chemical formula of water?,H2O,H2O,Chemical compounds data,Chemistry|Molecules,Name 8,Comments 8,
10-
What is the speed of light?,"299,792,458 meters per second","299,792,458 meters per second",Physics constants list,Physics|Speed of Light,Name 9,Comments 9,
11-
Who wrote 'Romeo and Juliet'?,William Shakespeare,William Shakespeare,Literary works list,Literature|Shakespeare,Name 10,Comments 10,
12-
What is the largest planet in our solar system?,Jupiter,Jupiter,Solar system data,Astronomy|Planets,Name 11,Comments 11,
13-
How many continents are there?,7,7,Continents list,Geography|Continents,Name 12,Comments 12,
14-
What is the boiling point of water?,100°C,100°C,Temperature data,Science|Temperature,Name 13,Comments 13,
15-
What is 15% of 200?,30,30,Basic arithmetic,Math|Percentages,Name 14,Comments 14,
16-
Who wrote 'Romeo and Juliet'?,William Shakespeare,William Shakespeare,Literary works list,Literature|Shakespeare,Name 15,Comments 15,
1+
input,actual_output,expected_output,retrieval_context,context,tools_called,expected_tools,name,comments,source_file
2+
What is the tallest mountain in the world?,Mount Everest,Mount Everest,Mountain heights list|Moutain list,Geography|Mountains,"[{""name"": ""mountain_lookup"", ""inputParameters"": { ""query"": ""tallest mountain"" }}]","[{""name"": ""mountain_lookup"", ""inputParameters"": { ""query"": ""tallest mountain"" }}]",Name 1,Comments 1,
3+
Solve 5 * 7,35,35,Basic arithmetic,Math|Multiplication,"[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""5 * 7"" }}]","[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""5 * 7"" }}]",Name 2,Comments 2,
4+
Define 'photosynthesis',Process by which green plants convert light energy into chemical energy,Process by which green plants convert light energy into chemical energy,Biology textbooks|Science API,Biology|Photosynthesis,"[{""name"": ""biology_lookup"", ""inputParameters"": { ""query"": ""photosynthesis"" }}]","[{""name"": ""biology_lookup"", ""inputParameters"": { ""query"": ""photosynthesis"" }}]",Name 3,Comments 3,
5+
What is the capital of France?,Paris,Paris,World capitals list,Geography|Capital Cities,"[{""name"": ""capitals_lookup"", ""inputParameters"": { ""query"": ""capital of France"" }}]","[{""name"": ""capitals_lookup"", ""inputParameters"": { ""query"": ""capital of France"" }}]",Name 4,Comments 4,
6+
What is the square root of 64?,8,8,Basic arithmetic,Math|Square Roots,"[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""sqrt(64)"" }}]","[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""sqrt(64)"" }}]",Name 5,Comments 5,
7+
What causes rain?,Rain is caused by the condensation of water vapor in the atmosphere.,Rain is caused by the condensation of water vapor in the atmosphere.,Weather patterns data,Science|Meteorology,"[{""name"": ""weather_lookup"", ""inputParameters"": { ""query"": ""rain causes"" }}]","[{""name"": ""weather_lookup"", ""inputParameters"": { ""query"": ""rain causes"" }}]",Name 6,Comments 6,
8+
Translate 'Hello' into Spanish.,Hola,Hola,Translation data,Language|Translation,"[{""name"": ""translator"", ""inputParameters"": { ""query"": ""Hello"", ""target_language"": ""Spanish"" }}]","[{""name"": ""translator"", ""inputParameters"": { ""query"": ""Hello"", ""target_language"": ""Spanish"" }}]",Name 7,Comments 7,
9+
What is the chemical formula of water?,H2O,H2O,Chemical compounds data,Chemistry|Molecules,"[{""name"": ""chemistry_lookup"", ""inputParameters"": { ""query"": ""chemical formula water"" }}]","[{""name"": ""chemistry_lookup"", ""inputParameters"": { ""query"": ""chemical formula water"" }}]",Name 8,Comments 8,
10+
What is the speed of light?,"299,792,458 meters per second","299,792,458 meters per second",Physics constants list,Physics|Speed of Light,"[{""name"": ""physics_lookup"", ""inputParameters"": { ""query"": ""speed of light"" }}]","[{""name"": ""physics_lookup"", ""inputParameters"": { ""query"": ""speed of light"" }}]",Name 9,Comments 9,
11+
Who wrote 'Romeo and Juliet'?,William Shakespeare,William Shakespeare,Literary works list,Literature|Shakespeare,"[{""name"": ""literature_lookup"", ""inputParameters"": { ""query"": ""Romeo and Juliet author"" }}]","[{""name"": ""literature_lookup"", ""inputParameters"": { ""query"": ""Romeo and Juliet author"" }}]",Name 10,Comments 10,
12+
What is the largest planet in our solar system?,Jupiter,Jupiter,Solar system data,Astronomy|Planets,"[{""name"": ""astronomy_lookup"", ""inputParameters"": { ""query"": ""largest planet"" }}]","[{""name"": ""astronomy_lookup"", ""inputParameters"": { ""query"": ""largest planet"" }}]",Name 11,Comments 11,
13+
How many continents are there?,7,7,Continents list,Geography|Continents,"[{""name"": ""geography_lookup"", ""inputParameters"": { ""query"": ""number of continents"" }}]","[{""name"": ""geography_lookup"", ""inputParameters"": { ""query"": ""number of continents"" }}]",Name 12,Comments 12,
14+
What is the boiling point of water?,100°C,100°C,Temperature data,Science|Temperature,"[{""name"": ""science_lookup"", ""inputParameters"": { ""query"": ""boiling point water"" }}]","[{""name"": ""science_lookup"", ""inputParameters"": { ""query"": ""boiling point water"" }}]",Name 13,Comments 13,
15+
What is 15% of 200?,30,30,Basic arithmetic,Math|Percentages,"[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""15% of 200"" }}]","[{""name"": ""calculator"", ""inputParameters"": { ""query"": ""15% of 200"" }}]",Name 14,Comments 14,
16+
Who wrote 'Romeo and Juliet'?,William Shakespeare,William Shakespeare,Literary works list,Literature|Shakespeare,"[{""name"": ""literature_lookup"", ""inputParameters"": { ""query"": ""Romeo and Juliet author"" }}]","[{""name"": ""literature_lookup"", ""inputParameters"": { ""query"": ""Romeo and Juliet author"" }}]",Name 15,Comments 15,

0 commit comments

Comments
 (0)