Large Language Models rely on diverse, well-annotated data to improve their accuracy and utility. Crowdsourcing provides scalable and affordable access to LLM data and the expertise needed for training and fine-tuning these models.
As an example, platforms such as Toloka crowdsource vetted human talent from around the world, often educated to degree level, to join their networks. This enables the collection of high quality annotated LLM data from global contributors, ensuring datasets are inclusive and representative of real-world language use. Toloka’s network also augments state-of-the-art AI & ML technologies with expert human feedback in sophisticated data pipelines. Its team has the expertise and experience to:
- Generate synthetic data from scratch, or validate a client’s pre-generated data at any stage.
- Select top-performing models with appropriate licenses tailored to a client’s needs.
- Develop complex data pipelines for processing raw internet-sourced data or proprietary datasets.
Toloka collaborates with data workers from 100+ countries speaking 40+ languages across 50+ knowledge domains and 120+ subdomains. In May 2025, Jeff Bezos’ investment firm, Bezos Expeditions, led a US$72 million funding round in Toloka. Let’s look at how it operates in some greater depth.
1. Enhancing Large Language Models (LLMs) Through Human Input
While LLMs are becoming more advanced, they still depend on human input for tasks like labeling, ranking responses, and validating outputs. Crowdsourcing bridges the gap between AI capabilities and human expertise, and can accelerate project completions.
Toloka integrates crowdsourcing workflows to refine models like GPT, BERT, and others. This is particularly beneficial in domain specific sectors such as healthcare, law and finance which are particularly difficult for AI. This is because they commonly require specific data due to the often complex requests or enquiries, compliance regulations, and the types of information that should be available only from qualified sources. Crowdsourcing enables easier and scalable collection of such specialized datasets for domain-specific LLMs.
Toloka’s founder and CEO, Olga Megorskaya. Image source: Toloka
For example, a Stanford University report in 2024 found that large language models used widely for medical assessments are often unable to back up claims. In the case of medical questions, Toloka’s Founder and CEO, Olga Megorskaya, says models should be trained through a process called alignment to avoid giving a diagnosis or medical advice, and to offer helpful information supported by medical references.
Two popular alignment techniques are RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization). In both approaches, the AI model outputs different responses and humans choose which one is better. This data is consumed by the alignment algorithm to train the model. These “human-in-the-loop” systems help refine outputs, identify errors, and continuously improve model performance.
2. Crowdsourcing for Multilingual and Low-Resource Language Support
Crowdsourcing taps into global pools of contributors to collect data for underrepresented languages, enabling the development of more inclusive and versatile language models. This can include language dialects, and languages spoken with an accent by non-native speakers.
Failing to cater for a number of individual minorities can ultimately exclude significant numbers of people from the benefits of LLM interaction. If these people are already using the internet less than the average level, it would make it more difficult to scrape the required data rather than use crowdsourcing. The problem will chase its own tail.
An example of this is the African language Swahili. It is no county’s national language, yet it is spoken in 14 countries by over 200 million people. Toloka carried out a project that required a network of Swahili speakers to judge the automatic translation of 15,000 questions and answers from English to Swahili. 4,000 low quality translations of questions and answers were rejected, and the final dataset was used to improve mT5, one of the top-performing multilingual language models for Swahili. The combination of automated translation with human validation offers a cost-efficient and scalable approach,
3. Cost-Effective Scalability with Quality Control
Some unconvinced potential users may be wary of the accuracy of results from crowdsourced non-specialists, though crowdsourcing offers a cost-effective way to scale data annotation and training efforts compared to in-house teams while platforms maintain quality control.
A simple method when crowdsourced annotators are labelling LLM data is to slip some known content in to what they are labelling. Their response to this data can be compared to what is already known about it, and annotators’ performance can then be judged on its accuracy.
An example of maintaining quality control is a project for a European clothing brand that wanted to introduce bodyscan technology to help their customers find the perfect size for each garment. Initial attempts to create a database were based on employees and their friends. However, the database was neither large enough or diverse enough to cover all the required body shapes.
Image source: Toloka
Members of Toloka’s crowd were asked to take photos of themselves while measuring 22 parameters of their body. They were from a wide range of countries to obtain diverse results, and were asked to submit the photos and measurements separately. There were some discrepancies that required verification, and the client’s team also checked the data and discarded incomplete or invalid measurements. By the end of the project, 500 complete sets of measurements were collected from the crowd.
Human assessments can not only be used for training data, but also for evaluating and benchmarking LLM performance.
4. Crowdsourced Data for Improved Context and Reduced Bias
Contextual understanding is critical for LLMs. Crowdsourced contributors provide nuanced labels and feedback, ensuring models understand context better.
Beyond instances containing words or phrases commonly recognized as vulgar, obscene or profane, toxicity in language, for example, can take the forms of sarcasm and hate speech, or direct personal attacks. How LLMs detect and deal with such incidents can be very important. A Toloka case study demonstrates the unique value of human input to put the intended meaning of words or phrases in to context. This example focuses on using the Ukrainian language – hardly mainstream, though achievable through harnessing the input of a crowd.
Bias in an LLM, on the other hand, can stem simply from a data imbalance to begin with. Crowdsourced efforts can input sufficient data to ensure gender-neutral and culturally inclusive responses.
5. Ethical Considerations in Crowdsourcing for LLMs
In addition to gender neutrality and cultural inclusiveness making LLM data reflect an external ethical balance, internal ethical practices in crowdsourcing should ensure fair compensation, contributor privacy, and transparent workflows.
Toloka’s approach to responsible AI is built on trust, security, excellence, and fairness. They integrate privacy principles at every stage of their processes, making data and identity protection a core consideration from the outset.
There are several kinds of tasks available on Toloka, which vary in difficulty, duration, and reward. Tasks can be chosen that suit a person’s interests, skills, and availability. Tasks can also be filtered by language, device, location, or other criteria.
Photo by Vardan Papikyan on Unsplash
Payment rates vary by the type of task. Labeling images, texts, audio, or video usually takes a few seconds or minutes to complete, and pays between $0.01 to $0.10 per task. Data collection tasks or completing surveys usually take from just a few minutes to hours to complete, and they pay from $0.10 to $10.00 per task.
Whatever the task, Toloka’s ethical practices ensure contributors are fairly compensated and data is responsibly handled. An example of fair compensation relates to the clothing brand I mentioned earlier that introduced bodyscan technology. Most people spent about 20 minutes taking measurements and submitting photos, which is longer than a typical task. Each participant received enhanced payment for submitting a set of measurements.
Key Takeaways
Crowdsourcing platforms like Toloka are transforming the development of Large Language Models (LLMs) by providing scalable, high-quality, and inclusive data solutions. Here are five compelling reasons to leverage Toloka:
High-Quality Human Input for Model Refinement
Toloka’s global network of vetted, degree-educated contributors delivers precise human feedback for tasks like data labeling, response ranking, and output validation, critical for refining LLMs like GPT and BERT. AI developers can use Toloka to enhance model accuracy in specialized domains like finance or law, where nuanced expertise is essential.
Inclusive Multilingual LLM Data for Global Reach
With contributors from 100+ countries speaking 40+ languages, Toloka supports LLM data collection for low-resource languages and dialects, such as Swahili (spoken by 200 million across 14 countries). Researchers can tap Toloka to build inclusive LLMs that serve diverse populations, reducing exclusion and expanding market potential.
Cost-Effective Scalability with Robust Quality Control
Toloka offers a cost-efficient alternative to in-house data annotation, scaling efforts through crowdsourcing while maintaining quality.
Improved Contextual Understanding and Bias Reduction
Toloka’s crowdsourced feedback provides nuanced labels to enhance LLM data’s contextual understanding and mitigate biases, such as toxicity (e.g., sarcasm, hate speech). The size of its network enables it to overcome imbalanced data bias. Developers can thus use Toloka to create fairer, more context-aware models, addressing critical ethical and performance challenges.
Ethical and Responsible AI Practices
Toloka prioritizes fair compensation, contributor privacy, and transparent workflows, aligning with responsible AI principles. In the bodyscan project, participants were paid extra for 20-minute tasks, reflecting equitable practices. With robust data protection integrated into its pipelines, Toloka builds trust. Organizations can partner with Toloka to uphold ethical standards, appealing to stakeholders and regulators in an era of heightened AI scrutiny.
0 Comments