Overview

Sotopia uses Redis (opens in a new tab) for two uses cases:

1. To store data including agent profiles, environment profiles, relationship profiles, and all of the interaction logs. This is useful for reproducibility and analysis.
2. To serve as a message broker between the Redis agent and frontend. This is useful for human user interface and other agents.

We mainly use Redis-OM (opens in a new tab) together with Pydantic (opens in a new tab) to interact with the Redis database, but also sometimes directly use the Redis-py (opens in a new tab) library for performance-critical tasks. Despite that Redis is 3rd party software, which is not included in the Sotopia package, we do realize that it is useful to include a short introduction to Redis in this documentation.

A 1-minute introduction to Redis

Redis is a key-value database, meaning that you can use it like a global Python dict. For example,

import redis
 
r = redis.Redis()
r.set("key", "value")
print(r.get("key")) # b'value'

Redis-OM, OM standing for Object Mapping, is a library that helps you easily use more complex data structures. For example, you can use it to store a person's profile like this:

from redis_om import JsonModel, Field
 
class Person(JsonModel):
    name: str = Field(index=True)
    age: int = Field(index=True, default_factory=lambda: 0)
 
# create a person profile
person = Person(name="Alice", age=30)
 
# save the person to the database
person.save()

You can then query the database like this:

# find all of the people in the database
for person in Person.find().all():
    print(person.name, person.age)
# find all of the people with the name "Alice"
for person in Person.find(name="Alice").all():
    print(person.name, person.age)

where JsonModel is a subclass of pydantic.BaseModel. If you are familiar with Pydantic, you should feel right at home with Redis-OM. Even if you are not, this is a very natural way to define data structures in Python.

Built-in data strcutures in Sotopia

In Sotopia, we have several built-in data structures that are used to store profiles, episodes, and other data, which you can used to interact with the Redis database. These include:

• AgentProfile
• RelationshipProfile
• EnvironmentProfile
• EpisodeLog
• AnnotationForEpisode
• EnvAgentComboStorage (opens in a new tab)

The data structures for message passing are:

• MessageTransaction (opens in a new tab)
• SessionTransaction (opens in a new tab)

Serialization and Deserialization

It is very easy to serialize any database structures to JSON or CSV.

Serialize simulated episodes

from sotopia.database import episodes_to_jsonl, EpisodeLog
 
episodes: list[EpisodeLog] = EpisodeLog.all()
 
episodes_to_jsonl(episodes, "episodes.jsonl")

Serialize env/agent profiles

from sotopia.database import environmentprofiles_to_jsonl, agentprofiles_to_jsonl
 
agent_profiles: list[AgentProfile] = AgentProfile.all()
 
environment_profiles: list[EnvironmentProfile] = EnvironmentProfile.all()
 
agentprofiles_to_jsonl(agent_profiles, "agent_profiles.jsonl")
environmentprofiles_to_jsonl(environment_profiles, "environment_profiles.jsonl")

Other utilities