user generated content Archives

Today, user-generated content like chat, private messaging, comments, images, and videos are all must-haves in an overstuffed market where user retention is critical to long-term success. Users love to share, and nothing draws a crowd like a crowd — and a crowd of happy, loyal, and welcoming users will always bring in more happy, loyal, and welcoming users.

But as we’ve seen all too often, there is risk involved when you have social features on your platform. You run the risk of users posting offensive content – like hate speech, NSFW images, and harassment – which can cause serious damage to your brand’s reputation.

That’s why understanding the risks when adding social features to your product are also critical to long-term success.

Here are four questions to consider when it comes to user-generated content on your platform.

1. How much risk is my brand willing to accept?
Every brand is different. Community demographic will usually be a major factor in determining your risk tolerance.

Communities with under-13 users in the US have to be COPPA compliant, so preventing them from sharing PII (personally identifiable information) is essential. Edtech platforms should be CIPA and FERPA compliant.

If your users are teens and 18+, you might be less risk-averse, but will still need to define your tolerance for high-risk content.

Consider your brand’s tone and history. Review your corporate guidelines to understand what your brand stands for. This is a great opportunity to define exactly what kind of an online community you want to create.

2. What type of high-risk content is most dangerous to my brand?
Try this exercise: Imagine that just one pornographic post was shared on your platform. How would it affect the brand? How would your audience react? How would your executive team respond? What would happen if the media/press found out?

What about hate speech? Sexual harassment? What is your brand’s definition of abuse or harassment? The better you can define these often vague terms, the better you will understand what kind of content you need to moderate.

3. How will I communicate my expectations to the community?
Don’t expect your users to automatically know what is and isn’t acceptable on your platform. Post your community guidelines where users can see them. And make sure users have to agree to your guidelines before they can post.

4. What content moderation tools and strategies can I leverage to protect my community?
We recommend taking a proactive instead of a reactive approach to managing risk and protecting your community. That means finding the right blend of pre- and post-moderation for your platform, while also using a mixture of automated artificial intelligence with real human moderation.

On top of these techniques, there are also different tools you can use to take a proactive approach, including in-house filters (read about the build internally vs buy externally debate), or content moderation solutions like Two Hat’s Community Sift (learn about the difference between a simple profanity filter and a content moderation tool).

Feeling overwhelmed?
While social features may be inherently risky, remember that they’re also inherently beneficial to your brand and your users. Whether you’re creating a new social platform or adding chat and images to your existing product, nothing engages and delights users more than being part of a positive and healthy online community.

And if you’re not sure where to start – we have good news.

Two Hat is currently offering a no-cost, no-obligation community audit. Our team of industry experts will examine your community, locate high-risk areas, and identify how we can help solve any moderation challenges.

It’s a unique opportunity to sit down with our Director of Community Trust & Safety to see how you can mitigate risk in your community.

To book your free audit, fill out the form below and we’ll reach out with next steps!

Community managers and online moderators know this all too well — moderating user-generated content can be draining.

You spend hours looking at text, usernames, and images. Content ranges from the mind-numbingly dull (false or accidental reports) to the emotionally devastating (discussions about suicide or abuse). Often, with a mountain of reports to sift through and a seemingly endless supply of content, it feels like you’ll never catch up.

The industry has long depended on user reports to moderate online content, without leveraging any safety layers in between. This approach has made for long, tedious workdays, and the inevitable emotional burnout that goes with it.

There has to be a better way.

Here are a few ideas you can leverage to keep your moderation team — and yourself — sane this year.

Something to keep in mind — we mostly talk about online games in this piece, but every technique is just as applicable to virtual worlds, social sharing apps, forums, and more.

1. Triage reports

Figuring out what is and isn’t a priority is one of the biggest challenges faced by moderation teams. Some companies receive up to 60,000 reports a day. In that sea of content, how can you possibly know what to review first? Inevitably, you’ll fall behind. And the longer you wait to review reported content and take action where needed, the less impact that action will have on future behavior.

But here’s the thing: Humans no longer need to do this kind of work. In the last few years, artificial intelligence has gotten a lot more… well, intelligent. That’s why you can start with an algorithm that can analyze reports as they come in, then move them into different queues based on risk level and priority. The great part is that algorithms can be trained on your community and your moderator’s actions. It’s not a one-size-fits-all approach.

Reports come in three varieties: No action needed, questionable, and you-better-deal-with-this.

Computers are great at identifying the easy stuff — the good (no action needed) — and the obviously bad (you-better-deal-with-this). Computers haven’t yet figured out how to make complex, nuanced decisions, which is where humans come in (the questionable). Human moderators belong in this grey, middle ground where our natural understanding of context and human behavior gives us an upper hand when making tough decisions.

Remember when Facebook removed the harrowing yet iconic photo of 9-year-old Kim Phuc fleeing a napalm attack in Vietnam? We can all understand how an algorithm would misunderstand the image and automatically remove it — and why a human reviewing the same picture would instead understand and consider the historical importance of an otherwise disturbing image.

That’s why it’s crucial that you leverage a twofold approach — humans and machines, working together, each playing to its strengths.

Players will report other players for no reason; it’s just human nature. So you will always have reports that don’t require action or human review. Whether you punish or restrict players for false reports is up to you (stay tuned for an upcoming blog where we explore this topic in further detail), but the end result is always the same — close the report and move on to the next one.

That’s wasted time that you and your team will never recover, regardless of how quickly you review, close, and take action on reports.

Want that time back? Try this approach:

Let the machine identify and automatically close reports that don’t require moderator review.
Identify and automatically take action on reports that clearly require moderator action. This can be done with a combination of automation and human review; many companies leverage auto-sanctions but still review high-risk content to ensure that no further action needs to be taken.
Identify content in the grey zone and escalate it to a queue for priority moderator review.

Of course, it doesn’t matter how smart AI is — building, testing, and tuning an algorithm takes precious time and resources — both of which are usually in short supply. That’s why we advocate for a “buy instead of build” approach (see Thinking of Building Your Own Chat Filter? 4 Reasons You’re Wasting Your Time! for more on this). Shop around to find the option that’s the best fit for your company.

2. Give moderators a break

Ever repeated a word so many times it lost all meaning? Spend too much time reviewing reports, and that’s exactly what can happen.

Even the most experienced and diligent moderators will eventually fall prey to one of two things:

Everything starts to look bad, or
Nothing looks bad

The more time you spend scanning chat or images, the more likely you are to see something that isn’t there or miss something important. The end result? Poor moderation and an unhappy community.

Break up the day with live moderation and community engagement (see below for more about these proactive techniques).
Have your team take turns working on reports so they are constantly being reviewed (if you want players to learn from their mistakes and change their behavior, it’s critical that you apply sanctions as close to the time of the offense as possible).
Ensure that moderators switch tasks every two hours to stay fresh, focused, and diligent.

3. Be proactive

Ultimately, what’s the best way to avoid having to review tens of thousands of reports? It’s simple: Don’t give players a reason to create reports in the first place. We don’t mean turn off the reporting function; it’s a critical piece in the moderation puzzle, and one of the best ways to empower your users. However, you can use a variety of proactive approaches that will curb the need to report.

Here are a few:

Use a chat/image filter

Until recently, content filters have had a bad rap in the industry. Many companies scoffed at the idea of blocking certain words or images because “the community can handle itself,” or they saw filters as promoting censorship.

Today, the conversation has changed — and we’re finally talking seriously about online abuse, harassment, and the very real damage certain words and phrases can have.

Not only that, companies have started to realize that unmoderated and unfiltered communities that turn toxic aren’t just unpleasant — they actually lose money. (Check out our case study about the connection between proactive moderation and user retention.)

Like we said earlier, computers are great at finding the best and worst content, as defined by your community guidelines. So why not use a chat filter to deal with the best and worst content in real time?

Think about it — how many fewer reports would be created if you simply didn’t allow players to use the words or phrases you’re taking action on already? If there’s nothing to report… there are no reports. Of course, you’ll never get rid of reports completely, and nor should you. Players should always be encouraged to report things that make them uncomfortable.

But if you could automatically prevent the worst of the worst from ever being posted, you can ensure a much healthier community for your users — and spare you and your hard-working team the headache of reviewing thousands of threatening chat lines and/or images in the first place.

You can also use your moderation software (and remember, we always recommend that you buy instead of build) to automatically sanction users who post abusive content (whether it’s seen by the community or not), as defined by your guidelines.

Speaking of automatic sanctions…

Warn players

We wrote about progressive sanctions in Five Moderation Workflows Proven to Decrease Workload. They’re a key component of any moderation strategy and will do wonders for decreasing reports.

From the same blog:

“Riot Games found that players who were clearly informed why their account was suspended — and provided with chat logs as backup — were 70% less likely to misbehave again.”

Consider this: What if you warned players — in real time — that if their behavior continued, they would be suspended? How many players would rethink their language? How many would keep posting abusive content, despite the warning?

The social networking site Kidzworld found that the Community Sift user reputation feature — which restricts and expands chat permissions based on player behavior — has encouraged positive behavior in their users.

Set aside time for live moderation

Even the smartest, most finely-tuned algorithm cannot compare to live moderation.

The more time spent following live chat and experiencing how the community interacts with each other in real time, the better and more effective your moderation practices will become.

Confused about a specific word or phrase you’re seeing in user reports? Monitor chat in real time for context. Concerned that a specific player may be targeting new players for abuse but haven’t been able to collect enough proof to take action? Take an hour or so to watch their behavior as it happens.

Live moderation is also an effective way to review the triggers you’ve set up to automatically close, sanction, or escalate content for review.

Engage with the community

What’s more fun than hanging out with your community — aside from playing the game, of course? ; )

There’s a reason you got into community management and moderation. You care about people. You’re passionate about your community and your product. You and your team of moderators are at your best when you’re interacting with players, experiencing their joy (and their frustration), and generally understanding what makes them tick.

So, in addition to live moderation, it’s critical that you and your team interact with the community and actively inspire positive, healthy communication. Sanctions work, but positive reinforcement works even better.

When you spend time with the community, you demonstrate that your moderation team is connected with players, engaged in the game, and above all, human.

Final thoughts

You’ll never eliminate all user reports. They’re a fundamental element of any moderation strategy and a key method of earning player trust.

But there’s no reason you should be forced to wade through thousands or even hundreds of reports a day.

There are techniques you and your team can leverage to mitigate the impact, including giving moderators a variety of tasks to prevent burnout and keep their minds sharp, using a proactive chat/image filter, and engaging with the community on a regular basis.

Two Hat empowers gaming and social platforms to foster healthy, engaged online communities.

Uniting cutting-edge AI with expert human review, our user-generated content filter and automated moderation software Community Sift has helped some of the biggest names in the industry protect their communities and brand, inspire user engagement, and decrease moderation workload.