This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
Monthly dumps are taking significantly longer to process right now because of how Reddit deals with large spammers. Spammers will make millions of comments to specific subreddits in the hopes it boosts their relevancy on search engines. This causes some days to have 20% or more of all comments being nothing but spam from specific accounts.
When monthly dumps are processed, there is a second pass made to check all missing ids to make sure as much data is collected as available. The issue is that when the first pass is done, there are millions of missing ids because these spam comments get removed from Reddit at some point.
It will be necessary to change the logic of the "fill the gaps" script so that it looks at the Pushshift API to see if the ids that are missing are from spammers.
With that said, May is almost done and June / July will start soon. Submissions are a bit easier to handle and will be brought up to date over the next 2-3 weeks.
Thanks!
Subreddit
Post Details
- Posted
- 5 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/pushshift/c...