Shop Buddy

Problem and Motivation

In the realm of artificial intelligence, the transformational impact is widespread, and one of the areas where this evolution is distinctly felt is the area of online shopping. Presently, users navigate through traditional websites using search queries and filters, often encountering a cumbersome and less-than-optimal buying experience.  

We found opportunities for value adds: 

1. Enhance user shopping experience through interactive chatbot leveraging Generative AI
2. Opportunity to Integrate large language model (LLM) in a B2C market

Our project centers on harnessing the power of Large Language Models (LLMs) to elevate and streamline the furniture shopping experience. This focus is particularly pertinent given the complex nature of furniture shopping, involving multi-dimensional and multi-model search challenges.

Minimum Viable Product (MVP)

Our MVP provides a personalized experience as it interacts with Natural Language through the power of LLM, fine tuned on industry specific furniture data and retrieves images, returns relevant images & actively changes search results based on user input through text -to-image CLIP model. We package it up with an interactive user interface that is easy to use by providing users with clickable furniture items to provide a smooth purchasing experience.

We aim to solve identified gaps, such that online shoppers will: 

1. Have a consistent and simple user experience irrespective of the furniture retailer
2. Be able to converse and express their needs to quickly find what they are looking for

Model Components & Observations

We employed GPT-3.5 + CLIP to leverage multi modal search and used prompt engineering with few-shot learning techniques to prompt the model. 

An example flow will be as such:

Stage 1:  The user inputs  "I have back pain sitting too long in the office. I like beige colored chairs" this will be feed into GPT-3.5. GPT-3.5 will output "user is looking for a beige office chair" this will be fed into the CLIP model. 

Stage 2: The model converts the product description into embeddings [ 1.2, 0.8...]. The embeddings of the product description is matched against the embeddings of each category, top 5 category kept (pre-filtering)

Stage 3: For our baseline model we compare the embeddings of the product description against the embeddings of the titles, top 5 images kept. In version 1 of our model in order to reduce run time and improve accuracy we used Chat GPT & pre-filtered data by matching on Category then outputting Title prior to CLIP image search. We compare the embeddings of the product description against the embeddings of the images & keep the top 5 images as output to the chatbot. In version 2 we kept it the same as version1, but evaluated on one subcategory. Final version of the model has been fine tuned on office furniture and evaluated on the office furniture subcategory.

In assessing the efficacy of our model's performance, we honed in on a key quantitative metric—the CLIP Score, which gauges cosine similarity between text and image features. Additionally, we focused on four qualitative metrics, comparing actual inputs with model outputs for Title Match (utilizing product titles), Description Match (leveraging product descriptions), Color and Category Match (using a combination of color and category inputs), and ChatGPT Input Match (employing ChatGPT to transform product titles and descriptions).

Observations:

1. Improvement in model performance: Fine-tuning efforts yielded significantly higher title match scores, showcasing the efficacy of our optimization strategies substantially between baseline model and version 3.
2. Trade-offs between performance and generalizability: Despite 61% improvement in scores, resource constraints prompted the decision to deploy Version 2 as the final model. Interestingly, however, the ChatGPT input match score exhibited a slight decrease compared to the model without fine-tuning. This suggests a nuanced outcome, hinting that the model may have become less generalizable in specific contexts. These observations underscore the intricacies involved in the modeling journey and highlight the need for careful consideration of resource constraints and trade-offs in the deployment decision-making process.

Key Learnings & Impact

Shop Buddy has made significant progress in addressing the needs of enhancing users furniture shopping experience through usage of natural language models. We believe that this will allow retailers to leverage the advancement of Gen AI to achieve overall growth in the market through providing buyer with customized and enhanced experience. Majority of users, who participated in our user study, have noted that they would buy using the chatbot  with promised search results.

Through the MVP development, the team has learned and gained skillsets on architecting a full end-to-end generative AI powered chatbot, integrating backend model results with frontend UI, addressing performance and latency issues when working with large data set, and overall product delivery from ideation to test. 

Future Work

In our ongoing pursuit of enhancing user experience and expanding the capabilities of our furniture shopping chatbot, we have identified key areas for future development. 

1. Enhancement of the user experience through added capabilities: This evolution aims to provide users with a more efficient and intuitive buying process directly through the chat interface, through integrating a in-bot purchase & payment capability into our chatbot. 
2. Expanding to other retail product domains:This expansion will involve broadening the chatbot's dataset to encompass a diverse array of retail categories, ensuring a robust and versatile shopping experience for users across various product domains.
3. Developing plugin integration options to integrate with existing systems: This will include exploring a range of plugins facilitating connectivity to email, calendar systems, and other communication channels, streamlining business processes and communication through the chatbot interface.
4. Continuous adoption of latest technology: The team is committed to continuously adopting technological advancements in the AI-driven retail solutions industry. This includes exploring customer experience chatbots by IBM WatsonX, drawing latest computer vision and material recognition model insights from academic papers to stay in the forefront.

Mitigating Potential Ethical Concerns

In acknowledging the ethical dimensions of deploying an AI-driven chatbot, we would like to proactively address potential issues to ensure transparency, privacy, and responsible use in the future.
 

1. Key concern involves users being unaware that they are interacting with a non-human entity. To mitigate this, we plan to incorporate a prominent label or warning in the user interface, providing clarity about the nature of the interaction and fostering transparency.
2. Handling of user data and search history. In response to this, our approach is straightforward—we do not intend to store any user information or search history. Furthermore, we are committed to developing a comprehensive privacy policy, ensuring users have clear insights into how their data is managed and protected in the future.
3. To preempt unintentional use cases or misuse of the chatbot, we are planning to integrate strategic safeguards. These include breaking points within the chatbot's capabilities, clearly communicated to users. For instance, incorporating limitations such as, "I can only answer questions about furniture," serves as a safeguard, preventing the chatbot from engaging in areas beyond its intended scope and minimizing the risk of unintended consequences.
4. Addressing the potential risk of users inadvertently entering personal information into the chatbot, we have implemented an opt-out mechanism for user data storage, specifically with respect to OpenAI. Users will have the ability to choose not to have their data stored, aligning with best practices for data privacy.

This holistic approach to mitigating potential ethical concerns underscores our commitment to responsible AI deployment and user protection.

Acknowledgments

We extend our sincere gratitude to our Capstone professors, Ramesh Sarukkai and Frederick Nugen. Their support has been instrumental in propelling our journey and realizing our ambitious goal of launching this product. Without their guidance and encouragement, we wouldn't have reached the significant milestones we've attained.

In addition, fellow classmate Max Kaufmann contributed to identifying ethical concerns & mitigating strategies.