DnD NPC AI

Problem and Motivation

Dungeons and Dragons is a very complex game which places considerable demands on the orchestrator of  that game, the DM or Dungeon Master. This person is responsible for the story or narrative of the game sessions, planning out the interactions between the player characters and the environment, to include NPCs or non-players characters. Finally the DM must also be ready to respond creatively and quickly to the actions of very unpredictable players. 

While there are many resources available for a new DM, instruction guides, reference books, and tutorial videos, these resources are referential in nature and therefore do not lend themselves too well to utilization in the middle of play, which is often exciting and dynamic. The DnD NPC AI seeks to address this by actively removing tasks from the DM and migrating them to an AI, in this case, the task of creating fun and unique dialogue between the player characters and the non-player characters. 

This task often involves spontaneously creating an entire persona and pretending to be this persona for as long as the players choose, a task which is daunting for the inexperienced DM and exhausting for the seasoned DM. The DnD NPC AI, seeks to increase the speed and efficacy by which the DM attempts to accomplish this task and to do so in a way that reduces the cognitive burden on the DM.

Data Science Approach

The system is a Retrieval Augmented Generation (RAG) system with three components:

1. The reference database is composed of a vector store (QDRant) and a Mongo database. The Vector store stores background information about each character as well as a log of previous interactions as a form of NPC memory while the Mongo Database manages the catalog of available characters.
2. The LLM, Llama 3 70bn Instruct, which uses the user prompt to pull relevant context out of the database in order to enrich the quality of the response, maintain alignment with the NPC persona, and provide the textual response.
3. The interface which uses the Discord application and its supporting APIs along with a FastAPI endpoints to allow for multiple user functions such as adding or modifying a character, querying a rules database, or enabling voice to text transcription.

These components are combined to create a seamless experience for the user.

Evaluation

Evaluation focused on two areas: 

1. The NPC chat bot’s ability to maintain immersion with its backstory and lore
2. The ability for rules lookup to produce correct answers

Evaluation of the chat bot's ability to maintain immersion was done by crafting a series of questions designed to assess this. These questions, along with information about the character and the character's responses to questions were evaluated by ChatGPT 40 and the number of violations we counted. 

Evaluation of the rules endpoint was more traditional, focusing on BLEU, ROGUE, and BERTScores, evaluated against gold standard answers, also generated by ChatGPT and verified by a human DnD player. Two types of answers were generated, simple answers which directly answer the question posed without any additional eloboration or examples, and complex answers which provided additional context and relevant edge cases pertaining to the rules question.

Key Learnings and Impact

1. LLM generates surprisingly detailed and immersive answers to player inputs and remains very aligned with provided context, however, LLM does not, without prompt or other interjection, ‘move the story’ along or take actions.
   1. This required additional features to allow the DM to guide the LLM and NPC towards decisive action when required for play.
2. LLM, with rules context, provides fairly good rules look-up, provided the rules reference contains the answer. Intuiting answers in the absence of a direct reference is still a limitation.

Testing trails with DnD play-groups showed an increase in the quality of conversations with players and NPCs. DMs do not have to think as quickly or be as creative in highly specific NPC interactions. Speed of service remains a challenge, especially in very dynamic interactions or with rapid conversations. Improving the way DMs and players interact with the service can improve this. 

With DnD’s growing popularity, a service such as this can make the game much more accessible to newer DMs and much more rewarding for newer players. Additionally, the ability to deploy a similar framework to purely digital environments (video games, etc.) is equally promising.