How We Manage Toxicity for Social Apps and Websites

At Two Hat, we believe the social internet is a positive place with unlimited potential. We also believe bullying and toxicity are causing harm to real people and causing irreparable damage to social products. That’s why we made Community Sift.

We work with leading game studios and social platforms to find and manage toxic behaviours in their communities. We do this in real-time, and (at the time of writing) process over 1 billion messages a month.

Some interesting facts about toxicity in online communities:

  • According to the Fiksu Index, the cost of acquiring a loyal user is now $4.23, making user acquisition one of the biggest costs to a game.
  • Player Behavior in Online Games research published by Riot Games indicates that “players are 320% more likely to quit, the more toxicity they experience.”

Toxicity hurts everyone:

  • An estimated 1% of a new community is toxic. If that is ignored, the best community members leave and toxicity can grow as high as 20%.
  • If a studio spends $1 million launching its game and a handful of toxic users send destructive messages, their investment is at risk.
  • Addressing the problem early will model what the community is for, and what is expected of future members, thus reducing future costs.
  • Behaviour does change. That’s why we’ve created responsive tools that adapt to changing trends and user behaviours. We believe people are coachable and have built our technology with this assumption.
  • Even existing communities see an immediate drop in toxicity with the addition of strong tools.

Here’s a little bit about what Community Sift can do to help:

  • More than a Filter: Unlike products that only look for profanity, we have over 1 million human-validated rules and multiple AI systems to seek out bullying, toxicity, racism, fraud, and more.
  • Emphasis on Reputation: Every user has a bad day. The real problem is users who are consistently damaging the community.
  • Reusable Common Sense: Instead of simple reg-ex or black/whitelist, we measure the severity on a spectrum, from extreme good to extreme bad. You can use the same rules but a different permission level for group chat vs. private chat and for one game vs. another.
  • Industry Veterans: Our team has made games with over 300 million users and managed a wide variety of communities across multiple languages. We are live and battle-tested on top titles, processing over 1 billion messages a month at the time of writing.

To install Community Sift, you have your backend servers make one simple API call for each message, and we handle all the complexity in our cloud.

When toxic behaviour is found, we can:

  • Hash out the negative parts of a message: e.g. *”####ed out message”*
  • Educate the user
  • Reward positive users who are consistently helping others
  • Automatically trigger a temporary mute for regular offenders
  • Escalate for internal review when certain conditions like “past history of toxicity” are met
  • Group toxic users on a server together to help protect new users
  • Provide daily stats, BI reports, and analytics

We’d love to show you how we can help protect your social product. Feel free to book a demo anytime.

Cyberbullying, Bullying, and Online Harassment Facts

What is Bullying?

Bullying is an unacceptable anti-social behavior. Bullying tactics can involve verbal abuse, physical abuse, and psychological harassment towards a victim.This destructive

This destructive behavior is learned by a bully from the negative influences that can come from others in their home environment, school, peer groups, and even from certain media exposure.

The good news is that since bullying is a learned behavior, it can also be unlearned.

What is Cyberbullying?

Cyberbullying is a form of bullying behavior that is inflicted online or by phone through text messages, email, instant messaging, chat rooms, website posts, or images sent to a cell phone or personal digital assistant.

Cyberbullying is distinguished from bullying by the nature of its anonymity, anytime accessibility, and punitive fears of the loss of the use of a digital device.

The Signs of Bullying?

The signs of bullying behavior can be things such as the desire to be in control, arrogance, impulsiveness, boastfulness, poor sportsmanship and a lack of empathy.

The signs of a victim of bullying may be a reluctance to go to school, mood changes, loss of appetite, torn clothes, and signs of physical abuse such as bruises.

Often bullying can occur at schools in certain high-risk areas such as the hallways, cafeterias, playgrounds, and buses.

Bullying Facts

  • Bullying is the most common form of violence in our society
  • Nearly 1 in 5 students have indicated that they had been bullied repeatedly over time (two to three times per month or more within the school semester)
  • Bullying impacts approximately 13 million students every year, and some 160,000 students stay home from school each day because of bullying
  • 52% of bullied victims will suffer incidents of repeated bullying for a duration of months at a time
  • 71% of bullying is witnessed by others
  • 61% of face to face bullying is reportedly done by males
  • 68% of cyberbullying is reportedly done by females

How to Stop Bullying

While it may seem like an insurmountable task to completely stop bullying, there are practical ways to counter bullying problems.

Some initiatives to take to help stop bullying and cyberbullying are:

  • Talk about bullying and harassment
  • Help others develop empathy, including those that are doing the bullying
  • Provide anti-bullying training
  • Don’t be a bystander, speak up, report, intervene
  • Don’t delay in acting
  • In school settings, parents need to be involved

Sandy Neeson, a licensed school counsellor at McLean Middle School in Fort Worth, Texas has said, “To successfully change bullying behavior, you must involve the whole school, from teachers and custodians to cafeteria workers and bus drivers.”

The key is that it takes an entire community to actively stop bullying.