Can clear community policies against harassment reduce its prevalence in a community? And what side effect (if any) do they have on freedom of expression?

In 2020, our team worked with moderators and community members of r/futurology to test the effect on newcomers of sticky comments that list community rules. This study was a replication of a 2016 study with r/science (you can read it here in PNAS). We now have results for r/futurology, as well as two other communities who tested the ideas in parallel.

In this thread, we're sharing the results to discuss the preliminary analysis. This is a space for you to ask questions, interpret the results, and discuss how (or if) these results should influence what the community does next.

I'll be available all day to field questions. We will compile what we learn from this conversation when writing up and submitting the results for peer review with an academic publication. Thanks!

Resources:

• Full Study Results: How well do harassment prevention interventions transfer between communities?
• Full data analysis
• Research page with full links

What we did with r/futurology

Starting in January 2020, our software observed when new posts were made and assigned discussions to receive either a sticky comment with the rules or no sticky comment at all. We then measured how many newcomer accounts commented and whether the first comment from newcomers was removed by moderators or not.

In r/futurology the message read:

​

Welcome to /r/Futurology! To maintain a healthy, vibrant community, comments will be removed if they are disrespectful, off-topic, or spread misinformation (rules). While thousands of people comment daily and follow the rules, mods do remove a few hundred comments per day. Replies to this announcement are auto-removed.

What we learned

In r/futurology, we did not observe an effect of rule posting on newcomer behavior.

Chart: https://citizensandtech.org/wp-content/uploads/2022/08/2022-08-22-newcomer-norm-compliance-replication-576x576.png

Across all subreddits on average, posting the rules increased the chance that first-time commenters would follow the rules. However, r/science was the only community with a statistically-significant effect both times. With r/ffxiv, we think the reason is that rule violations are fairly rare. With r/futurology, the cause might be the sample size issue or something else. We are eager to hear your ideas in the thread.

What effect did the sticky comment have on newcomer participation? While newcomer comments increased in the first r/science study in 2016, we did not find an affect on levels of newcomer participation in the follow-up studies. We discuss possible reasons for this in the post.

Finally, we found that the effect on moderator workload depended largely on whether the intervention increased newcomer participation or not.

Caveats to discuss

Due to a software glitch, the study didn't have as large a sample size in r/futurology as our projections indicated we needed—we had to remove 3,600 posts (26%) of the posts from the study. We're working to see if we can gain access to historical mod logs to fill in this gap.

Note on Ethics

Note: The study was reviewed by the moderators of the subreddit and approved by the Princeton and then the Cornell University ethics boards (Cornell protocol #1909009059). If you have any concern, we encourage you to ask it below or reach out to us directly. If you do not feel comfortable doing so, you can contact the Cornell Institutional Review Board here.

Please Share Your Questions, Reactions, and Ideas

MANY THANKS to everyone in the subreddit who supported this research, and for your patience as we worked to set it up and write up the results during COVID!

I'll be here all day to field questions and discuss the results, so do please share any reactions and ideas.