The current Achilles Hill of the Web is no longer its speed or amount of content, neither is it the ease in which new content gets created and circulated. Where it falls short is in content quality and suitability from the perspective of its consumers. Consumers vary, and net neutrality is king, but for any given consumer, most of the Web is noise, if not outright harmful. This problem is not just about user experience. Pollution of the public discourse is a threat to democracy, and dilution of the subjective value of web content to its users is a threat to its overall usefulness and future. Our phenomenal success in making content so easy to produce and distribute, whatever this content is, plants the seeds of the biggest threat to both the web’s usability, and to some of the societal norms that we appreciate. Artificial Intelligence, if deployed properly, can help us overcome the difficulties posed by the overall information overload and quality-dilution that our success in implementing the Web has caused.

There are many reasons for content to be irrelevant to some of the users, and those reasons are always subjective. Even machine-generated content that we may classify as “noise” or “propaganda” may not be considered as such by some people who may enjoy consuming it. Furthermore, the moment we architect judgmental models into the fabric of the Web, we put an inevitable end to much of what the Web is and should remain being. Our solution should be focused on enabling the user to overcome information overload, coping with the majority of web content being irrelevant; irrelevant by that user’s own standard.

There is a precedent for this situation. Decades ago, we coped with a simplified version of the “mostly irrelevant content” problem. Also then, the problem was highlighted by the high accessibility we put into our Internet protocols, and also then we solved the problem using (a simple version of) machine learning. The e-mail transfer protocol, SMTP, was created without much security built into it. Most notably, anyone can send an e-mail appearing to come from just about anyone else. Soon enough, the e-mail system was swamped by spam messages, which at one time accounted for almost 93% of all e-mail traffic. This was an existential threat to e-mail. Fortunately, Bayes filtering was invented, and implemented in spamassassin and other tools at different levels. This type of filtering, unlike simple static rule-based filtering, learns over time what types of genuine e-mails the user gets, and after not too long of standing corrected by the user and the infrastructure operator, reaches an impressive filtering accuracy. It is only thanks to this mechanism (and others that followed) that we no longer consider spam mail as a problem worth worrying about.

It is time to expand this model dramatically. The challenge now is not just e-mail but web content in general, we do not need to filter spam from ham (using spam filtering jargon) but to rank the relevance and appropriateness of content, and we need to account for different users that have entirely different definitions of content relevance or appropriateness, which even they cannot always describe. If we do this properly, we stand a chance at making today’s massive amounts of diverse web content as attention-worthy to its consumers as web content used to be when it was carefully produced and consumed using the amount- and diversity-limiting waterfall model.

If I keep on thinking about this further, then I will add future posts with future thoughts...