Evaluate and Fine-Tune Your AI models

Our suite of subjective evaluation tests can further tune your AI models with accurate human in the loop workflows.  If you care about the Quality of Experience for your virtual assistants, TTS models, machine translation engines or chatbots, partner with us today.

Get Started

MOS Testing

Obtain a Mean Opinion Score for a single speech, text or image stimulus

MUSHRA Testing

Evaluate and score multiple speech, text or image stimuli

ABX Testing

Compares 2 choices of a random sample against 2 reference outputs to identify detectable differences

Pronunciation Validation

A more objective evaluation designed to evaluate text-to-speech

Evaluation Quality Guarantee

Given subjective nature of opinion-based work, the same level of DefinedCrowd quality cannot be guaranteed, however multiple quality metrics are used to ensure evaluations are performed and validated, including the naturalness, trustworthiness, and likeability of your AI model.

Spammer control

IP location-based blocking of contributors

Inter-evaluator checks

Agreement score calculations using Pearson Correlation Coefficient

Standard deviation

Answer variety thresholds based on standard deviation and evaluations per contributor

Real time audits

Answer pattern detection for suspicious behavior

Success Stories

For this project, Mastercard’s R&D Labs needed unique, multi-lingual text data that covered 20 designated payment

Keeping a nation’s lights on means constantly inspecting electricity poles for damage. Before EDP partnered with

With the rise of voice technology, this leading global provider of audio equipment wanted to develop an automatic speech recognition (ASR) model

When a global electronics maker came to DefinedCrowd with the goal of building more inclusive facial-recognition

Smart companies see the pile of unstructured text floating through the digital realm as a strategic goldmine

In the arms race for speech-enabled technologies, systems that can support the widest user base will win out

Visionary companies like Amazon are leveraging sentiment analysis models to dig beyond surface-level understandings