Quora: What are the different ways to moderate content?

Written by

There are five different approaches to User-Generated Content (UGC) moderation:

  • Pre-moderate all content
  • Post-moderate all content
  • Crowdsourced (user reports)
  • 100% computer-automated
  • 100% human review

Each option has its merits and its drawbacks. But as with most things, the best method lies somewhere in between — a mixture of all five techniques.

Let’s take a look at the pros and cons of your different options.

Pre-moderate all content

  • Pro: You can be fairly certain that nothing inappropriate will end up in your community; you know you have human eyes on all content.
  • Con: Time and resource-consuming; subject to human error; does not happen in real time, and can be frustrating for users who expect to see their posts immediately.

Post-moderate all content

  • Pro: Users can post and experience content in real-time.
  • Con: Once risky content is posted, the damage is done; puts the burden on the community as it usually involves a lot of crowdsourcing and user reports.

Crowdsourcing/user reports

  • Pro: Gives your community a sense of ownership; people are good at finding subtle language.
  • Con: Similar to pre-moderating all content, once threatening content is posted, it’s already had its desired effect, regardless of whether it’s removed; forces the community to police itself.

100% computer-automated

  • Pro: Computers are great at identifying the worst and best content; automation frees up your moderation team to engage with the community.
  • Con: Computers aren’t great at identifying gray areas and making tough decisions.

100% human review

  • Pro: Humans are good at making tough decisions about nuanced topics; moderators become highly attuned to community sentiment.
  • Con: Humans burn out easily; not a scalable solution; reviewing disturbing content can have an adverse effect on moderator’s health and wellness.
    So, if all five options have valid pros and cons, what’s the solution? In our experience, the most effective technique uses a blend of both pre- and post-moderation, human review, and user reports, in tandem with some level of automation.

The first step is to nail down your community guidelines. Social products that don’t clearly define their standards from the very beginning have a hard time enforcing them as they scale up. Twitter is a cautionary tale for all of us, as we witness their current struggles with moderation. They launched the platform without the tools to enforce their (admittedly fuzzy) guidelines, and the company is facing a very public backlash because of it.

Consider your stance on the following:

  • Bullying: How do you define bullying? What behavior constitutes bullying in your community?
  • Profanity: Do you block all swear words or only the worst obscenities? Do you allow acronyms like WTF?
  • Hate speech: How do you define hate speech? Do you allow racial epithets if they’re used in a historical context? Do you allow discussions about religion or politics?
  • Suicide/Self-harm: Do you filter language related to suicide or self-harm, or do you allow it? Is their a difference between a user saying “I want to kill myself,” “You should kill yourself,” and “Please don’t kill yourself”?
  • PII (Personally Identifiable Information): Do you encourage users to use their real names, or does your community prefer anonymity? Can users share email addresses, phone numbers, and links to their profiles on other social networks? If your community is under-13 and in the US, you may be subject to COPPA.

Different factors will determine your guidelines, but the most important things to consider are:

  • The nature of your product. Is it a battle game? A forum to share family recipes? A messaging app?
  • Your target demographic. Are users over or under 13? Are portions of the experience age-gated? Is it marketed towards adults-only?

Once you’ve decided on community guidelines, you can start to build your moderation workflow. First, you’ll need to find the right software. There are plenty of content filters and moderation tools on the market, but in our experience, Community Sift is the best.

A high-risk content detection system designed specifically for social products, Community Sift works alongside moderation teams to automatically identify threatening UGC in real time. It’s built to detect and block the worst of the worst (as defined by your community guidelines), so your users and moderators don’t ever have to see it. There’s no need to force your moderation team to review disturbing content that a computer algorithm can be trained to recognize in a fraction of a second. Community Sift also allows you to move content into queues for human review, and automate actions (like player bans) based on triggers.

Once you’ve tuned the system to meet your community’s unique needs, you can create your workflows.

You may want to pre-moderate some content, even with a content filter running in the background. If your product is targeted at under-13 users, as an added layer of human protection, you might pre-moderate anything that the filter doesn’t classify as high-risk. Or maybe you route all content flagged as high-risk (extreme bullying, hate speech, rape threats, etc) into queues for moderators to review. For older communities, you may not require any pre-moderation and instead depend on user reports for any post-moderation work.

With an automated content detection system in place, you give your moderators their time back to do the tough, human stuff, like dealing with calls for help and reviewing user reports.

Another piece of the moderation puzzle is addressing negative user behavior. We recommend using automation, with the severity increasing with each offense. Techniques include warning users when they’ve posted high-risk content, and muting or banning their accounts for a short period. Users who persist can eventually lose their accounts. Again, the process and severity here will vary based on your product and demographic. The key is to have a consistent, well-thought-out process from the very beginning.

You will also want to ensure that you have a straightforward and accessible process for users to report offensive behavior. Don’t bury the report option, and make sure that you provide a variety of report tags to select from, like bullying, hate speech, sharing PII, etc. This will make it much easier for your moderation team to prioritize which reports they review first.

Ok, so moderation is a lot of work. It requires patience and dedication and a strong passion for community-building. But it doesn’t have to be hard if you leverage the right tools and the right techniques. And it’s highly rewarding, in the end. After all, what’s better than shaping a positive, healthy, creative, and engaged community in your social product? It’s the ultimate goal, and ultimately, it’s an attainable one — when you do it right.

 

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: What is the single biggest problem on the internet?

Written by

It has to be the proliferation of dangerous content. For good or for evil, many social networks and online communities are built around the concept of total anonymity — the separation of our (socially, ethically, and legally) accountable offline identities from our (too often hedonistic, id-driven, and highly manufactured) online identities.

People have always behaved badly. That’s not pessimism or fatalism; it’s just the truth. We are not perfect; often we are good, but just as often we indulge our darkest desires, even if they hurt other people.

And so with the advent of a virtual space where accountability is all too often non-existent, the darkest parts of the real world—harassment, rape threats, child abuse — all moved onto the internet. In the “real world” (an increasingly amorphous concept, but that’s a topic for another day), we are generally held accountable for our behavior, whereas online we are responsible only to ourselves. And sometimes, we cannot be trusted.

Facebook Live is a recent example. When used to share, engage, connect, and tell stories, it’s a beautiful tool. It’s benign online disinhibition at its best. But when it’s used to live stream murder and sexual assault — that’s toxic online disinhibition at its worst. And in the case of that sexual assault, at least 40 people watched it happen in real time, and not one of them reported it.

How did this happen?

It started with cyberbullying. We associate bullying with the playground, and since those of us who make the rules — adults — are far removed from the playground, we forget just how much schoolyard bullying can hurt. So from the beginning social networks have allowed bullying to flourish. Bullying became harassment, which became threats, which became hate speech, and so on, and so forth. We’ve tolerated and normalized bad behavior so long that it’s built into the framework of the internet. It’s no surprise that 40 people watched a live video of a 15-year-old girl being assaulted, and did nothing. It’s not difficult to trace a direct line from consequence-free rape threats to actual, live rape.

When social networks operate without a safety net, everyone gets hurt.

The good thing is, more and more sites are realizing that they have a social, ethical, and (potentially) legal obligation to moderate content. It won’t be easy — as Facebook has discovered, live streaming videos are a huge challenge for moderators — but it’s necessary. There are products out there — like Community Sift — that are designed specifically to detect and remove high-risk content in real-time.

In 2017, we have an opportunity to reshape the internet. The conversation has already begun. Hopefully, we’ll get it right this time.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: Why do people say things on the internet which they wouldn’t say in the real world?

Written by

Way back in 2004 (only 13 years ago but several lifetimes in internet years), a Professor of Psychology at Rider University named John Suler wrote a paper called The Online Disinhibition Effect. In it, he identifies the two kinds of online disinhibition:

Benign disinhibition. We’re more likely to open up, show vulnerability, and share our deepest fears. We help others, and we give willingly to strangers on sites like GoFundMe and Kickstarter.

Toxic disinhibition. We’re more likely to harass, abuse, and threaten others when we can’t see their face. We indulge our darkest desires. We hurt people because it’s easy.

Suler identified eight ways in which the internet facilitates both benign and toxic disinhibition. Let’s look at three of them:

Anonymity. Have you ever visited an unfamiliar city and been intoxicated by the fact that no one knew you? You could become anyone you wanted; you could do anything. That kind of anonymity is rarely available in our real lives. Think about how you’re perceived by your family, friends, and co-workers. How often do you have the opportunity to indulge in unexpected — and potentially unwanted — thoughts, opinions, and activities?

Anonymity is a cloak. It allows us to become someone else (for better or worse), if only for the brief time that we’re online. If we’re unkind in our real lives, sometimes we’ll indulge in a bit of kindness online. And if we typically keep our opinions to ourselves, we often shout them all the louder on the internet.

Invisibility. Anonymity is a cloak that renders us—and the people we interact with—invisible. And when we don’t have to look someone in the eye it’s much, much easier to indulge our worst instincts.

“…the opportunity to be physically invisible amplifies the disinhibition effect… Seeing a frown, a shaking head, a sigh, a bored expression, and many other subtle and not so subtle signs of disapproval or indifference can inhibit what people are willing to express…”

Solipsistic Introjection & Dissociative Imagination. When we’re online, it feels like we exist only in our imagination, and the people we talk to are simply voices in our heads. And where do we feel most comfortable saying the kinds of things that we’re too scared to normally say? That’s right—in our heads, where it’s safe.

Just like retreating into our imagination, visiting the internet can be an escape from the overwhelming responsibilities of the real world. Once we’ve associated the internet with the “non-real” world, it’s much easier to say those things we wouldn’t say in real life.

“Online text communication can evolve into an introjected psychological tapestry in which a person’s mind weaves these fantasy role plays, usually unconsciously and with considerable disinhibition.”

The internet has enriched our lives in so many ways. We’re smarter (every single piece of information ever recorded can be accessed on your phone — think about that) and more connected (how many social networks do you belong to?) than ever.

We’re also dumber (how often do you mindlessly scroll through Facebook without actually reading anything?) and more isolated (we’re connected, but how well do we really know each other?)

Given that dichotomy, it makes sense that the internet brings out both the best and the worst in us. Benign disinhibition brings us together — and toxic disinhibition rips us apart.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: How can you reinforce and reward positive behavior in an online community?

Written by

Online communities have unlimited potential to be forces for positive change.

Too often we focus on the negative aspects of online communities. How many articles have been written about online toxicity and rampant trolling? It’s an important topic — and one we should never shy away from discussing — but for all the toxicity in the online world, there are many acts of kindness and generosity that go unlooked.

There are a few steps that Community Managers can take to reinforce and reward positive behavior in their communities:

Promote and reinforce community guidelines. Before you can begin to champion positive behavior, ensure that it’s clearly outlined in your code of conduct. It’s not enough to say that you don’t allow harassment; if you want to prevent abuse, you have to provide a clear definition of what abuse actually entails.

A study was conducted to measure the effects of boundaries on children’s play. In one playground, students were provided with a vast play area, but no fences. They remained clustered around their teacher, unsure how far they could roam, uncertain of appropriate behavior. In another playground, children were given the same amount of space to play in, but with one key difference—a fence was placed around the perimeter. In the fenced playground, the children confidently spread out to the edges of the space, free to play and explore within the allotted space.

The conclusion? We need boundaries. Limitations provide us with a sense of security. If we know how far we can roam, we’ll stride right up to that fence.

Online communities are the playgrounds of the 21st century—even adult communities. Place fences around your playground, and watch your community thrive.

The flipside of providing boundaries/building fences is that some people will not only stride right up to the fence, they’ll kick it until it falls over. (Something tells us this metaphor is getting out of our control… ) When community members choose not to follow community guidelines and engage in dangerous behavior like harassment, abuse, and threats, it’s imperative that you take action. Taking action doesn’t have to be Draconian. There are innovative techniques that go beyond just banning users.

Some communities have experimented with displaying warning messages to users who are about to post harmful content. Riot Games has conducted fascinating research on this topic. They found that positive in-game messaging reduced offensive language by 62%.

For users who repeatedly publish dangerous content, an escalated ban system can be useful. On their first offence, send them a warning message. On their second, mute them. On their third, temporarily ban their account, and so on.

Every community has to design a moderation flow that works best for them.

Harness the power of user reputation and behavior-based triggers. These techniques use features that are unique to Community Sift, but they’re still valuable tools.

Toxic users tend to leave signatures behind. They may have their good days, but most days are bad—and they’re pretty consistently bad. On the whole, thee users tend to use the same language and indulge in the same antisocial behavior from one session to the next.

The same goes for positive users. They might have a bad day now and then; maybe they drop the stray F-bomb. But all in all, most sessions are positive, healthy, and in line with your community guidelines.

What if you could easily identify your most negative and most positive users in real time? And what if you could measure their behavior over time, instead of a single play session? With Community Sift, all players start out neutral, since we haven’t identified their consistent behavior yet. Over time, the more they post low-risk content, the more “trusted” they become. Trusted users are subject to a less restrictive content filter, allowing them more expressivity and freedom. Untrusted users are given a more restrictive content filter, limiting their ability to manipulate the system.

You can choose to let users know if their chat permissions have been opened up or restricted, thereby letting your most positive users know that their behavior will be rewarded.

Publicly celebrate positive users. Community managers and moderators should go out of their way to call out users who exhibit positive behavior. For a forum or comments section, that could mean upvoting posts or commenting on posts. In a chat game, that could look like publicly thanking positive users, or even providing in-game rewards like items or currency for players who follow guidelines.

We believe that everyone should be free to share without fear of harassment or abuse. We think that most people tend to agree. But there’s more to stopping online threats than just identifying the most dangerous content and taking action on the most negative users. We have to recognize and reward positive users as well.

Originally published on Quora 

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: Is Facebook doing enough to prevent suicide on its platform?

Written by

Is 2017 the year we see a kinder, safer web? It’s starting to look like it. On February 16th, Mark Zuckerberg published his mission statement, Building Global Communities. Two weeks later on March 1st Facebook rolled out new suicide prevention tools.

It’s great to see a big player like Facebook take on a challenging subject in such a big way. They understand that to create a safe and thriving community, it’s always better to be proactive than to do nothing. Facebook is demonstrating its commitment to creating a safe, supportive, and inclusive community with these new tools. We expect to see more and more features like this in the months to come.

Suicide is one of the biggest issues facing social networks today. The internet is full of self-injury and suicidal language, images, and videos. If we want to build communities where users feel safe and find a place they can call home, then we’re also responsible for ensuring that at-risk users are given help and support when they need it most.

Facebook has over 1.86 billion monthly active users, so they have access to data and resources that other companies can only dream of. Every community deserves to be protected from dangerous content. Is there anything smaller companies can do to keep their users safe?

After years in the industry studying high-risk, dangerous content we have unique insight into this issue.

There are a few things we’ve learned about self-injury and suicidal language:

Using AI to build an automation workflow is crucial. Suicide happens in real time, so we can’t afford mistakes or reactions after-the-fact. If you can identify suicidal language as it happens, you can also use automation to push messages of hope, provide suicide and crisis hotline numbers, and suggest other mental health resources. With their new features, Facebook has taken a huge, bold step in this direction.

Suicidal language is complex. If you want to identify suicidal language, you need a system that recognizes nuance, looks for hidden (unnatural) meaning and understands context and user reputation. There is a huge difference between a user saying “I am going to kill myself” versus “You should go kill yourself.” One is a cry for help, and the other is bullying. So it’s vital that your system learns the difference because they require two very different responses.

Think about all the different ways someone could spell the word “suicide.” Does your system read l337 5p34k? What if “suicide” is hidden inside a string of random letters?

Chris Priebe, CEO and founder of Two Hat Security (creator of Community Sift) wrote a response to Mark’s initial manifesto. In it he wrote:

When it comes to cyber-bullying, hate-speech, and suicide the stakes are too high for the current state of art in NLP [Natural Language Processing].

At Two Hat Security, we’ve spent five years building a unique expert system that learns new rules through machine learning, aided by human intelligence. We use an automated feedback loop with trending phrases to update rules and respond in real-time. We call this approach Unnatural Language Processing (uNLP).

When it comes to suicide and other high-risk topics, we aren’t satisfied with traditional AI algorithms that are only 90-95% accurate. We believe in continual improvement. When lives are at stake, you don’t get to rest on your laurels.
Suicide is connected to bullying and harassment. If you want to keep your community safe, you have to deal with all high-risk content. Community guidelines are great, but you need cutting-edge technology to back them up.

We’ve identified a behavioral flow that shows a direct link between cyberbullying/harassment and self-injury/suicide. When users are bullied, they are more likely to turn to suicidal thoughts and self-injuring behavior. It’s important that you filter cyberbullying in your product to prevent vulnerable users from getting caught in a vicious cycle.

While Facebook is doing its part, we want to ensure that all communities have the tools to protect their most vulnerable users. If you’re concerned about high-risk content in your community, we can help. Our content filter and moderation engine Community Sift is highly tuned to identify sensitive content like suicide and self-injury language.

We believe that everyone should be able to share online without being worried about harassed or threatened. Our goal has always been to remove bullying and other high-risk content from the internet. A big part of that goal involves helping online communities keep their most vulnerable users safe and supported. Suicide is such a sensitive and meaningful issue, so we want to extend our gratitude to Mark and all of the product managers at Facebook for taking a stand.

Here’s to hoping that more social networks will follow.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: What can social networks do to provide safer spaces for women?

Written by

For many women, logging onto social media is inherently dangerous. Online communities are notoriously hostile towards women, with women in the public eye—journalists, bloggers, and performers—often facing the worst abuse. But abuse is not just the province of the famous. Nearly every woman who has ever expressed an opinion online has had these experiences: Rape threats. Death threats. Harassment. Sometimes, even their children are targeted.

In the last few years, we’ve seen many well-documented cases of ongoing, targeted harassment of women online. Lindy West. Anita Sarkeesian. Leslie Jones. These women were once famous for their talent and success. Now their names are synonymous with online abuse of the worst kind.

And today we add a new woman to the list: Allie Rose-Marie Leost. An animator for EA Labs, her social media accounts were targeted this weekend in a campaign of online harassment. A blog post misidentified her as the lead animator for Mass Effect: Andromeda, and blamed her for the main character’s awkward facial animations. Turns out, Leost never even worked on Mass Effect: Andromeda. And yet she was forced to spend a weekend defending herself against baseless, crude, and sexually violent attacks from strangers.

Clearly, social media has a problem, and it’s not going away anytime soon. And it’s been happening for years.

A 2014 report by the Pew Research Center found that:

Young women, those 18-24, experience certain severe types of harassment at disproportionately high levels: 26% of these young women have been stalked online, and 25% were the target of online sexual harassment.

Young Women Experience Particulary Severe Form of Harassment

We don’t want to discount the harassment and abuse that men experience online, in particular in gaming communities. This issue affects all genders. However, there is an additional level of violence and vitriol directed at women. And it almost always includes threats of sexual violence. Women are also more likely to be doxxed, the practice of sharing someone else’s personal information online without their consent.

So, what can social networks do to provide safer spaces for women?

First, they need to make clear in their community guidelines that harassment, abuse, and threats are unacceptable —regardless of whether they’re directed at a man or a woman. For too long social networks have adopted a “free speech at all costs” approach to community building. If open communities want to flourish, they have to define where free speech ends, and accountability begins.

Then, social networks need to employ moderation strategies that:

Prevent abuse in real time. Social networks cannot only depend on moderators or users to find and remove harassment as it happens. Not only does that put undue stress on the community to police itself, it also ignores the fundamental problem—when a woman receives a rape threat, the damage is already done, regardless of how quickly it’s removed from her feed.

The best option is to stop abuse in real time, which means finding the right content filter. Text classification is faster and more accurate than it’s ever been, thanks to recent advances in artificial intelligence, machine learning, and Natural Language Processing (NLP).

Our expert system uses a cutting-edge blend of human ingenuity and automation to identify and filter the worst content in real time. People make the rules, and the system implements them.

When it comes to dangerous content like abuse and rape threats, we decided that traditional NLP wasn’t accurate enough. Community Sift uses Unnatural Language Processing (uNLP) to find the hidden, “unnatural” meaning. Any system can identify the word “rape,” but a determined user will always find a way around the obvious. The system also needs to identify the l337 5p34k version of r4p3, the backwards variant, and the threat hidden in a string of random text.

Take action on bad actors in real time. It’s critical that community guidelines are reinforced. Most people will change their behavior once they know it’s unacceptable. And if they don’t, social networks can take more severe action, including temporary or permanent bans. Again, automation is critical here. Companies can use the same content filter tool to automatically warn, mute, or suspend accounts as soon as they post abusive content.

Encourage users to report offensive content. Content filters are great at finding the worst stuff and allowing the best. Automation does the easy work. But there will always be content in between that requires human review. It’s essential that social networks provide accessible, user-friendly reporting tools for objectionable content. Reported content should be funnelled into prioritised queues based on content type. Moderators can then review the most potentially dangerous content and take appropriate action.

Social networks will probably never stop users from attempting to harass women with rape or death threats. It’s built into our culture, although we can hope for a change in the future. But they can do something right now—leverage the latest, smartest technology to identify abusive language in real time.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: How big a problem are bullying and harassment of pre-teens and teenagers on social media?

Written by

The numbers indicate that cyberbullying and harassment are huge problems for young people on social media. A 2016 report from the Cyberbullying Research Center indicates that 33.8% of students between 12 and 17 were victims of cyberbullying in their lifetime. Conversely, 11.5% of students between 12 and 17 indicated that they had engaged in cyberbullying in their lifetime.

Cyberbullying is different from “traditional” bullying in that it happens 24/7. For victims, there is no escape. It’s not confined to school or the playground. Kids and teens connect through social media, so for many, there is no option to simply go offline.

Even more troubling is the connection between cyberbullying and child exploitation. At Two Hat Security, we’ve identified a cycle in which child predators groom young victims, who are tricked into taking explicit photos which are then shared online; this leads to bullying and harassment from peers and strangers. Finally, the victim suffers from depression, engages in self-harm, and sometimes — tragically — commits suicide. It’s a heartbreaking cycle.

Cyberbullying and online harassment are profoundly dangerous and alarming behaviors with real, often severe and sometimes fatal, consequences for victims.

Social media platforms have options, though. AI-based text and image filters like Community Sift are the first lines of defense against cyberbullying. Purposeful, focused moderation of User Generated Content (UGC) is the next step. And finally, education and honest, open discussions about the effects of cyberbullying on real victims is crucial. The more we talk about it, the more comfortable victims will feel speaking out about their experiences.

Originally published on Quora, featured in Huffington Post and Forbes

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: Does it make sense for media companies to disallow comments on articles?

Written by

It’s not hard to understand why more and more media companies are inclined to turn off comments. If you’ve spent any time reading the comments section on many websites, you’re bound to run into hate speech, vitriol, and abuse. It can be overwhelming and highly unpleasant. But the thing is, even though it feels like they’re everywhere, hate speech, vitriol, and abuse are only present in a tiny percentage of comments. Do the math, and you find that thoughtful, reasonable comments are the norm. Unfortunately, toxic voices almost always drown out healthy voices.

But it doesn’t have to be that way.

The path of least resistance is tempting. It’s easy to turn off comments — it’s a quick fix, and it always works. But there is a hidden cost. When companies remove comments, they send a powerful message to their best users: Your voice doesn’t matter. After all, users who post comments are engaged, they’re interested, and they’re active. If they feel compelled to leave a comment, they will probably also feel compelled to return, read more articles, and leave more comments. Shouldn’t media companies cater to those users, instead of the minority?

Traditionally, most companies approach comment moderation in one of two ways, both of which are ineffective and inefficient:

  • Pre-moderation. Costly and time-consuming, pre-moderating everything requires a large team of moderators. As companies scale up, it can become impossible to review every comment before it’s posted.
  • Crowdsourcing. A band-aid solution that doesn’t address the bigger problem. When companies depend on users to report the worst content, they force their best users to become de facto moderators. Engaged and enthusiastic users shouldn’t have to see hate speech and harassment. They should be protected from it.

I’ve written before about techniques to help build a community of users who give high-quality comments. The most important technique? Proactive moderation.

My company Two Hat Security has been training and tuning AI since 2012 using multiple unique data sets, including comments sections, online games, and social networks. In our experience, proactive moderation uses a blend of AI-powered automation, human review, real-time user feedback, and crowdsourcing.

It’s a balancing act that combines what computers do best (finding harmful content and taking action on users in real-time) and what humans do best (reviewing and reporting complex content). Skim the dangerous content — things like hate speech, harassment, and rape threats — off the top using a finely-tuned filter that identifies and removes it in real-time. That way no one has to see the worst comments. You can even customize the system to warn users when they’re about to post dangerous content. Then, your (much smaller and more efficient) team of moderators can review reported comments, and even monitor comments as they’re posted for anything objectionable that slips through the cracks.

Comments section don’t have to be the darkest places on the internet. Media companies have a choice — they can continue to let the angriest, loudest, and most hateful voices drown out the majority, or they can give their best users a platform for discussion and debate.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: How do you build a community of users that give high-quality comments on a website?

Written by

There are a few key steps you can take to build a community that encourages high-quality comments:

  1. Know your community. What kind of articles will you be publishing? Will you cover controversial topics that are likely to elicit passionate responses? What demographic are you targeting? Once you know who will be posting (and what topics they will be posting about), you can start to…
  2. Think long and hard about community guidelines. If you know your community, you can create guidelines to protect. Be clear about your policies. If you allow profanity but not bullying, define bullying for your audience. If you allow racial slurs within the context of a historical article but not when they’re directed at another user, make sure it’s explained in your policy guide.
  3. Build a comprehensive moderation strategy. Visit the comments section of most websites, and you’re bound to walk away with a skewed — and highly unpleasant — view of humanity. Toxic voices will always drown out healthy voices. But it doesn’t have to be that way. If you’re using a blend of smart computer automation and human review to moderate comments, you can build a process that works for your unique community.
  4. Engage with your best users. Who doesn’t appreciate a good shout-out? Encouraging high-quality content can go a long way towards fostering a healthy community. Give your moderators the time to upvote, call out, or comment on quality posts. If you’ve done the first three steps, your moderators will have time to do what people do best, and what computers will likely never do—interact with users and engage them emotionally.

This is by no means an exhaustive list. Your community will grow and change over time, so you may have to adjust your policies as your audience changes. You will probably make mistakes and have to course-correct your moderation strategy. But if you start with a solid baseline, you will serve your audience well.

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required


Quora: What moderation techniques work best for social networks?

Written by

Moderation is a delicate art. It can take some real finesse to get it right. Every community is different and requires different techniques. But there are a few guiding principles that work for just about every product, from social networks to online games to forums.

Something to consider as you build your moderation strategy:

  • You have the power to shape the community.
  • Words have real consequences.

They may seem unconnected, but they’re profoundly linked. When creating a set of community guidelines and deciding how you will communicate and support them, you’re acknowledging that your community deserves the best experience possible, free of abuse, threats, and harassment. There is an old assumption that trolls and toxicity are inevitable by-products of the great social experiment that is the Internet, but that doesn’t have to be true. With the right techniques—and technology—you can build a healthy, thriving community.

First, it’s crucial that you set your community guidelines and display them in an area of your app or website that is readily available.

Some things to consider when setting guidelines:

  • The age/demographic of your community. If you’re in the US, and your community is marketed towards users under 13, by law you have to abide by the Children’s Online Privacy Protection Rule (COPPA). The EU has similar regulations under the new General Data Protection Rule (GDPR). In addition to regulating how you store Personally Identifiable Information(PII) on your platform, these laws also affect what kinds of information users can share with each other.
  • Know exactly where you stand on topics like profanity and sexting. It’s easy to take a stand on the really bad stuff like rape threats and hate speech. The trickier part is deciding where you draw the line with less dangerous subjects like swearing. Again, the age and demographic of your community will play into this. What is your community’s resilience level? Young audiences will likely need stricter policies, while mature audiences might be able to handle a more permissive atmosphere.
  • Ensure that your moderation team has an extensive policy guide to refer to. This will help avoid misunderstandings and errors when taking actions on user’s accounts. If your moderators don’t know your guidelines, how can you expect the community to follow them?

Then, decide how you are going to moderate content. Your best option is to leverage software that combines AI (Artificial Intelligence) with HI (Human Intelligence). Machine learning has taken AI to a new level in the last few years, so it just makes sense to take advantage of recent advances in technology. But you always need human moderators as well. The complex algorithms powering AI are excellent at some things, like identifying high-risk content (hate speech bullying, abuse, and threats). Humans are uniquely suited to more subtle tasks, like reviewing nuanced content and reaching out to users who have posted cries for help.

Many companies decide to build content moderation software in-house, but it can be expensive, complex, and time-consuming to design and maintain. Luckily, there are existing moderation tools on the market.

Full disclosure: My company Two Hat Security makes two AI-powered content moderation tools that were built to identify and remove high-risk content. Sift Ninja is ideal for startups and new products that are just establishing an audience.Community Sift is an enterprise-level solution for bigger products.

Once you’ve chosen a tool that meets your needs, you can build out the appropriate workflows for your moderators.

Start with these basic techniques:

  • Automatically filter content that doesn’t meet your guidelines. Why force your users to see content that you don’t allow? With AI-powered automation, you can filter the riskiest content in real time.
  • Automatically escalate dangerous content (excessive bullying, cries for help, and grooming) to queues for your moderators to review.
  • Automatically take action on users based on their behavior. Warn, mute, or ban users who don’t follow the guidelines. It’s not about punishment—Riot Games found that users who are given immediate feedback are far less likely to re-offend:

“When players were informed only of what kind of behavior had landed them in trouble, 50% did not misbehave in a way that would warrant another punishment over the next three months.”

  • Give users a tool to report objectionable content. Moderators can then review the content and determine if further action is required.

Building community is the fun part of launching a new social product. What kind of community do you want? Once you know the answer, you can get started. Draft your community guidelines, know how you will reinforce them, and invest in a moderation system that uses a blend of artificial and human intelligence. And once the hard stuff is out of the way—have fun, and enjoy the ride.  : )

Originally published on Quora

Want more articles like this? Subscribe to our newsletter and never miss an update!

* indicates required