App stores include a vast amount of user feedback in the form of app ratings and reviews. Research and recently also tool vendors have proposed data mining and data analytics solutions to leverage this feedback to developers and analysts, e.g., for extracting requirements-related information, monitoring the opinions of users on apps’ features, or informing release decisions. Research also showed that positive feedback improves apps’ downloads and sales figures and is thus fundamental to the success of apps.

As a side effect, a market for fake, incentivized app reviews emerged with yet unclear consequences for developers, app users, and app store operators. Fake reviews are prohibited in popular app stores, such as in Google Play or Apple App Store. Even governmental competition authorities started taking actions against companies using fake reviews to embellish their apps. For instance, the Canadian telecommunication provider Bell was fined $1.25 million for faking positive reviews to their apps.

Our journal-first paper reports on a study of fake reviews, their providers, characteristics, and how accurately they can be automatically detected. (1) We identified and conducted disguised questionnaires with 43 fake review providers and studied their review policies to understand their strategies and offers. We found that developers buy reviews to relatively expensive prices of a few dollars or deal with reviews in exchange portals (i.e., “I will write a positive review to your app if you do the same for my app”). The questionnaires and policies revealed that fake reviews are written to look authentic and are hard to recognize by humans. (2) By identifying and comparing 60,000 fake reviews with 62 million reviews from the Apple App Store, we found significant differences, e.g., between the corresponding apps, reviewers, rating distribution, and frequency. (3) This inspired the development of a simple classifier to automatically detect potential fake reviews in app stores. Based on the identified differences between fake and regular reviews, we developed, trained, fine-tuned, and compared multiple supervised machine learning approaches. (4) To have a more realistic setting of how our classifier can perform in practice, we conducted an in-the-wild experiment by varying the skewness of our dataset. On a labelled and imbalanced dataset, including one-tenth of fake reviews, as reported in other domains, our classifier achieved a recall of 91% and an AUC/ROC value of 98%.

We publicly share our gold-standard fake reviews dataset to enable the development of more accurate classifiers for identifying fake reviews or fake reviews candidates.

We discuss our findings and their impact on software engineering, app users, and app store operators. Although information extracted from app reviews is getting increasingly integrated into the requirements engineering process, none of the previous works in the research area app store analysis have considered fake reviews and their implications. By applying an existing app store analysis approach, we showed that requirements-related feedback, in the form of bug reports and feature requests, is included in both fake and regular reviews. Ignoring fake reviews might lead to wrong assumptions about actual user needs. Not optimal decisions for future development might be drawn too. Moreover, fake reviews might damage the integrity of app stores. Recently, Google highlighted the negative effects of fake reviews in an official statement and explicitly asked developers not to buy, and users not to provide fake reviews.