Scaling Content Moderation for Massive Datasets – Communications of the ACM

The volume of UGC creates the first and arguably most straightforward challenge. Thousands of new pieces of content are uploaded every minute across platforms. On YouTube alone, more than 500 hours of video are uploaded each minute. With so much content pouring in, it’s nearly impossible for human moderators to keep pace. The amount of content makes it more likely that dangerous material will fall through the cracks, whether hate speech, graphic violence, or disinformation.

UGC takes a variety of forms, ranging from text and images to video, live streaming, and even audio. Each of these demands different moderation strategies, and some are harder to evaluate than others. For instance, video editing techniques can be used to manipulate content in ways that make it difficult for automated systems to detect misleading or harmful material. Videos can include violence or hate speech, but also incorporate contextual subtleties such as sarcasm, which are hard for automated systems to identify with accuracy.

The appropriateness and meaning of a given piece of content are very much dependent on context. An apparently harmless piece of text or image can be dangerous in a different context, or based on where it is shared. For example, a term that is harmless in one culture can be considered offensive in another. In addition, humor and satire can easily be misjudged as hurtful speech by algorithms.

Content that is uploaded onto platforms by a world audience means moderators have to be in touch with the social, linguistic, and cultural dynamics of different parts of the world. Huge differences in what is acceptable or offensive from one country or culture to the next is likely. It becomes challenging to create a universal strategy for content moderation.

In the age of rapid-paced digital life, content needs to be moderated live to prevent the propagation of poisonous or deceptive material. If indecent content is left viewable for some time, then it can end up causing physical harm, for example, by motivating violence or promulgating fake information. It is not only a challenge to delete content speedily, but without interrupting people’s experiences as well.

Platforms also have to navigate a tangled mess of legal rules, which differ from country to country. In the E.U., for instance, the General Data Protection Regulation (GDPR) governs how user data has to be treated, while in the U.S., platforms have to obey Section 230 of the Communications Decency Act, which gives platforms immunity for content generated by users, but also invites platforms to act against illegal content.

Traditional Content Moderation Approaches

In the past, content moderation has involved the use of human moderators to examine and pass judgment on user-submitted content. Such moderators usually work under strict criteria established by the platform, flagging content that contravenes community guidelines or legal requirements. Human moderators can offer context awareness and nuanced judgment, but are limited by scalability factors. With billions of content pieces uploaded daily, human moderators are not in any position to keep up with the numbers. Automation is where they can fit in.

Scaling Content Moderation with Artificial Intelligence

The solution to scaling content moderation in a world of UGC overload is the use of artificial intelligence (AI) and machine learning (ML). These technologies enable platforms to automate a large part of content moderation and content creation, from detecting explicit images to identifying hate speech in text. AI-based systems can process huge datasets much more efficiently than human moderators.

AI can be taught to identify objectionable content in images and videos. For instance, computer vision methods can identify nudity, violence, or other explicit material. In the case of videos, AI can scan both the visual and audio aspects, identifying objectionable language, acts, or imagery. Facebook and Instagram utilize AI to automatically identify and delete such content that goes against their policies, i.e., hate speech or explicit content.

NLP models can assist in identifying offensive language in text posts, comments, and messages. The systems scan text to detect hate speech, harassment, and disinformation. For instance, AI can mark text containing racist or sexist language or identify cases of cyberbullying. NLP can also interpret sentiment, which assists in determining the tone of posts and messages.

AI is able to look beyond single pieces of content to measure and monitor user behavior over time. Through identification of patterns of abuse or violation, AI systems are able to identify users who consistently carry out harmful actions, like spreading misinformation or hate. AI systems are also able to identify abnormal activity, like the bulk sharing of spam or propaganda.

Certain platforms are using AI to forecast which content is most likely to break community guidelines even before it is uploaded. Through examination of past trends and data, AI algorithms can forecast possible breaches and mark them for inspection to ensure such content is not posted at all.

Although AI and ML are full of potential, they are by no means flawless. The accuracy of automated systems can be inconsistent, particularly with context, linguistic subtlety, or picking up on more subtle forms of injurious content. AI systems have the potential to unfairly censor material, misunderstand innocuous content, or miss more advanced forms of hate speech or disinformation.

Hybrid Models: Combining Human Moderators with AI

Because AI has limitations, most platforms are embracing hybrid approaches to content moderation that take advantage of the power of both automatic systems and human intervention. These hybrid approaches leave the majority of content moderation to AI, where it detects and flags obviously toxic content. It is human moderators who check out that flagged content and make contextual assessments, as well as dealing with edge cases that were not detected by the AI.

For instance, YouTube’s policy on content moderation employs AI in flagging the videos that it deems unacceptable, including videos with hate speech, graphic content, or misinformation. Such flagged videos are then examined by human moderators before they make a final ruling on removing the content or not. The hybrid model utilizes the scale offered by AI without sacrificing the fineness and correctness that only human moderators could offer.

Ethical Considerations and Transparency

As platforms expand their content moderation with AI, ethical considerations take center stage. Among the largest concerns is the risk of biased algorithms. Because AI models are trained on past data, they can inherit and amplify biases in the historical data. For example, if an AI system is trained on a dataset that disproportionately marks up content from a specific demographic or cultural group, it can unduly censor that group’s content. In addition, AI might find it difficult to identify content in non-standard formats, like satire, memes, or cultural allusions, and thus less reserved about removing such content.

Transparency is another key problem. Users must have the right to know how content moderation happens and have the ability to appeal if they think their content has been incorrectly marked or removed. Most platforms are still not forthcoming about how their content moderation infrastructure operates and how data is being used to train AI algorithms. Greater transparency along these lines will assist in earning the trust of users and keeping moderation fair.

Future Directions: The Evolving Landscape of Content Moderation

Looking ahead, several trends are likely to shape the future of content moderation:

AI and ML will increasingly dominate the role of content moderation. With the advances in these technologies, their capacities for correctly identifying problematic content and contextually-informed decision-making will increase, allowing for diminishing human moderator involvement.

Sites could begin to provide users with more control over the content they are shown. Personalized content moderation systems where users can define their own content filters according to their sensitivities and preferences might become more prevalent.

As decentralized technologies such as blockchain become more popular, platforms might consider alternative models of content moderation that rely on community governance and user-driven moderation styles.

Governments are also paying closer attention to how platforms moderate content, with some demands for greater regulation and responsibility. We might witness the creation of global content moderation standards to help ensure AI systems are free from bias and transparent in nature.

Conclusion

Scaling content moderation for enormous collections of user-generated content is perhaps the most urgent problem of the digital era. The volume, diversity, and complexity of content uploaded in a single minute are so vast that human moderators cannot possibly keep pace. But with the assistance of AI, machine learning, and hybrid models, platforms can start to address these issues, taking down harmful content in real time and keeping users safe while preserving freedom of expression.

Although these technologies have come a long way, practical and ethical hurdles still exist. The secret to success in the future will be balancing automated systems with human input, making sure that content moderation is fair, transparent, and effective. As UGC on the Internet continues to grow, so too must our strategies for content moderation, making the online world an open, safe, and respectful space for communication.

Alex Tray is a system administrator and cybersecurity consultant with 10 years of experience. He is currently self-employed as a cybersecurity consultant and as a freelance writer.