A Linguistic Experiment on the Yandex.Toloka Crowdsourcing Platform: An Evaluation of Its Research Capacity

The paper aims at analyzing online recruitment for linguistic experiments via Yandex.Toloka crowdsourcing platform in comparison to data collection via social media postings. In particular, we recruited respondents to complete the same acceptability judgment experiment on the position of negative pronouns in Russian subject control infinitive clauses. We use the results from the combined sample as control and investigate how well the two samples reflect the findings.
The demographics of the two samples differ slightly from the Yandex.Toloka sample in being more diverse in terms of educational background and place of origin. However, the employment relations result in a higher number of outliers in comparison to respondents who provide responses voluntarily. The experimental results show that the negative pronoun is licensed in the base position and necessarily moved to a preverbal position. The significant results across the two samples were almost indistinguishable, however,
a difference was found in the judgments coherence of a sample. Using resampling simulations, we evaluated the statistical power for the two samples and found that for Yandex.Toloka a larger sample size is required to achieve the 100% rate of statistical detectability. Finally, we analyzed the respondents’ individual consistency with regard to the combined sample results and with regard to their own responses. The data shows that the distribution of different groups of respondents is the same for the two samples with the majority of speakers being moderately consistent. We conclude that the power issues that arise within Yandex.Toloka can be solved by increasing the number of participants.