apprAIse
Problem & Motivation
Mission: To democratize access to expert-level vehicle evaluations through cutting-edge data science tools. By combining computer vision and machine learning, we aim to enhance transparency, negotiation power, and trust in second-hand car transactions. This ensures that buyers and sellers can make informed decisions with confidence and ease.
The process of buying or selling a used car often involves websites like Kelley Blue Book or Edmunds, where users self-report their vehicle’s condition and specifications to estimate its market value.This approach has significant drawbacks:
- Subjectivity of self-reporting: Users may unintentionally misrepresent their vehicle’s condition due to a lack of expertise.
- Limited precision: Assessments are often constrained by the user’s knowledge and ability to describe the car accurately.
- Time-intensive process: Filling out lengthy forms and conducting manual evaluations can be cumbersome.
- This results in an imbalanced market where sellers may have more power due to information asymmetry.
Data Source & Data Science Approach
Data
- Car Image Data
- Source: CARDD dataset
- Description: A comprehensive dataset containing various types of car damages and annotated bounding boxes.
- Severity Data
- Source: Source: CARDD dataset
- Description: We manually annotated all 4,000 images with a severity level ranging from 1 (most severe) to 4 (least severe).
- Audio Data
- Source: Scraped from YouTube Shorts featuring five distinct engine noise types (e.g., normal operation, bad alternator, bad transmission).
- Processing: Converted .wav files to spectrograms and trained a YOLOv8 model for classification.
Models
- Damage Detection Model
- YOLOv8 object detection model trained on the car image data to identify and localize damages on vehicle surfaces.
- Severity Classification Model
- YOLOv8 classification model trained on our manually annotated severity levels to dynamically assess and categorize the severity and extent of detected vehicle damages.
- Audio Classification Model
- YOLOv8 classification model trained on spectrograms of engine sounds to detect issues such as squealing belts or bad transmissions.
Architecture
The process begins with users entering their VIN, mileage, and zip code, then uploading four car images. This data flows through our frontend, hosted on AWS Amplify, to our API Gateway, which connects to two Lambda functions.
The first Lambda function, VINDetection, uses AWS Textract to extract the VIN directly from a scanned image, saving users the hassle of typing it in. The second Lambda function houses our core models. Here, the uploaded images are processed by our YOLOv8 damage detection model, which identifies damages and highlights them with bounding boxes. Simultaneously, a Yolov8 classification model evaluates the severity of the damages—determining if it’s a minor scratch or something more significant.
Next, the VIN, mileage, and zip code are passed to our Market Value API, which provides a price range for the user’s car. Combining this with the damage severity, we predict the car’s valuation. This data then flows into our LLM, Claude 3 Haiku, for a short report for the user on how to improve the car’s valuation. We also integrate a depreciation model, stored in S3, to generate a forecast graph showing how the car’s value changes over time.
Finally, all outputs—the images, valuation, the report, and depreciation graph—are bundled into a JSON file and sent back to the user via the API Gateway.
Evaluation
- Damage Detection Model
- Initially evaluated with mAP (73%) then we switched to precision and recall (76%) for their practical relevance—precision minimizes false positives, and recall ensures all damages are detected.
- Severity Classification Model
- Achieved 88% accuracy
- We chose this metric as it directly measures how well the model assigns the correct damage severity level (1-4).
- Audio Classification Model
- Achieved 83% accuracy
- Used this metric to assess its ability to categorize engine sounds into distinct classes despite variability in audio quality.
Key Learnings & Impact
Learnings/Challenges:
- With more data, we would have been able to develop an even more accurate model with more powerful crack, dent, scratch predictions. Our audio dataset was created from scratch by scraping online videos where quality differed greatly.
- The accuracy of our model hinges on the reliability and relevance of our training data. With many dynamic market factors and external influences across the supply chain, it is challenging to assess the precision of our output.
- We noticed that things such as trees reflecting off of car windows resulted in false positives. Additionally, lighting conditions can obscure damage and lead to false negatives.
Impact
apprAIse transforms the second-hand car market by:
- Automating vehicle damage assessments with high accuracy.
- Streamlining the valuation process with real-time evaluations.
- Promoting transparency and fairness in used car transactions.
Acknowledgements
- Prof Fred Nugen and Prof Morgan Ames