This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
19
[Change Log] This is the public change log for the Pushshift API
Author Summary
Stuck_In_the_Matrix is
in
Change Log
Post Body
| Type | Date | Description |
|---|---|---|
| Feature | 2019-03-27 | Recent Comment scores will now start updating after a 24 hour delay. |
| Feature | 2019-03-27 | Histograms by Score are now possible. |
| Feature | 2019-03-28 | Recent submission scores will now start updating after a 24 hour delay. |
| Bug Fix | 2019-03-29 | Commas in the q or title parameter would cause the query to crash. Commas now work as expected. |
| Maintenance | 2019-03-30 | Force merging segments in index rc_delta to increase query efficiency. This will expunge deleted documents and reduce the number of segments that have to be searched during queries using this index. |
| Feature | 2019-03-31 | Added four new aggregations: "author:score:avg", "author:score:sum", "subreddit:score:avg", "subreddit:score:sum" |
| Bug Fix | 2019-03-31 | Fetching ids for submissions was restricted to 10 results (https://www.reddit.com/r/pushshift/comments/b7s50t) -- Max is now 1,000. Limit parameter is not needed when using this endpoint. |
| Maintenance | 2019-03-31 | The rc_delta index is too large (it spans from October 01, 2017 to January 2019). This index is being slowly reindexed to multiple indexes with the naming convention of rc_yyyy-mm. Backporting Reddit's new gilding methodology (silver, gold, plat) to work consistently with older data. This will take approximately 4-5 days to complete. |
| Feature | 2019-03-31 | Adding the ability to filter by author_cakeday (part of the previous reindexing that was mentioned). This has been added to the mapping but is not yet live. |
| Feature | 2019-03-31 | Adding the ability to filter comments by the comment author's creation date. Also adding the Author's creation date to comment objects. This will allow filtering comments based on how old the author's account is. |
| Feature | 2019-03-31 | Adding the field "updated_utc" to comment and submission mappings. This will give the most recent time that the document was updated within Elasticsearch and will be helpful for ranking objects by score, etc. |
| Feature | 2019-03-31 | Added "author_cakeday" to current comment index so that all new comments ingested have correct mapping and support for this field. (curl -s -XPUT es2:9200/rc_delta2/_mapping/comments -d '{"properties":{"author_cakeday":{"type":"boolean"}}}') |
| Feature | 2019-03-31 | Added "author_cakeday" to the list of accepted boolean parameters so that comments can now be filtered by author_cakeday. Example: http://api.pushshift.io/reddit/comment/search/?after=24h&author_cakeday=true |
| Feature | 2019-03-31 | Added "author_cakeday" to the list of accepted boolean parameters for submissions. Also updated the submission mapping to support filtering on this field. (curl -s -XPUT es2:9200/rs_deltad/_mapping/submissions -d '{"properties":{"author_cakeday":{"type":"boolean"}}}') Example query: http://api.pushshift.io/reddit/submission/search/?after=1d&author_cakeday=true |
| Feature | 2019-03-31 | Added "author_flair_text" to the comment mapping so that all new comments are filterable by this field. Aggregations are also supported on this parameter. |
| Feature | 2019-03-31 | Added "is_submitter" field to comment mapping to filter by comments made by the submission submitter. Added support for API to filter based on this parameter. Example:http://api.pushshift.io/reddit/comment/search/?after=1h&is_submitter=true -- Submissions where a large percentage of comments are made by the submitter are usually always spam. |
| Maintenance | 2019-04-01 | Added a new normalizer to the comment mapping (my_normalizer) |
| Feature | 2019-04-01 | Added ability to filter on the "distinguished" parameter for comments. For example, to filter comments where comment is distinguished by a moderator: http://api.pushshift.io/reddit/comment/search/?after=1h&distinguished=moderator |
| Announcement | 2019-04-01 | Sold Pushshift to Facebook -- Now called Faceshift. |
| Maintenance | 2019-04-01 | Moving the main ES indexing code (that feeds from the ingest) from Perl to Python. |
| Feature | 2019-04-01 | Added aggregation capability for distinguished field. Example: http://api.pushshift.io/reddit/comment/search/?aggs=distinguished&after=24h&size=0 |
| Maintenance | 2019-04-02 | Moved the Ingest feed to Secondary DB due to a drive issue on the Primary DB |
| Maintenance | 2019-04-02 | Installed Elasticsearch 7.0 rc1 to a test server to start testing existing code base on newest ES version |
| Bug Fix | 2019-04-02 | Fixed "/pushshift" Slack bot issue (403_client_error) issue (due to a broken new code path that was released on 2019-03-29) |
| Status | 2019-04-02 | February monthly comment ingest is now at the halfway point and should be available by Wednesday of next week. |
| Planned Outage | 2019-04-04 | Partial outage from 1 AM ET until 6 AM ET. Results prior to January 20, 2019 may have duplicates or other issues during this time. |
| Feature | 2019-04-03 | New Endpoint to look up authors. Example: https://api.pushshift.io/reddit/author/lookup/?author=stuck_in_the_matrix,automoderator -- The max number of authors per request is capped at 1,000. If more than 1,000 authors are sent, only the first 1,000 are processed. |
| Feature | 2019-04-04 | Added the parameters "since" and "until" to officially replace "after" and "before" -- The previous parameters will still be accepted so that existing code bases don't break. These two new parameters will be the "official" parameters going forward. |
| Maintenance | 2019-04-04 | Primary DB storage is now critically low (99% full with 25 GB remaining out of the original 3 TB of space). This will be upgraded within the next week. This Postgres database holds all real-time ingest data as a secondary backup to the ES indices. |
| Maintenance | 2019-04-04 | Upgraded the Google Drive account to allow for up to one petabyte of backup storage. |
| Planned Outage | 2019-04-07 | Partial outage from 1 AM ET until 6 AM ET. Results prior to February 1, 2019 may have duplicates or other issues during this time. |
| Outage Ended | 2019-04-07 | The planned outage has concluded (1:30 AM ET). Please let me know if you discover any issues. |
| Feature | 2019-04-08 | Added the following quarantined subreddits to the ingest: braincels, cringeanarchy, subforwhitepeopleonly, theredpill |
| Status | 2019-04-12 | February comments are 90% complete. A dump should be available on Sunday or Monday at the latest. |
| Feature | 2019-04-12 | Expanded the list of tracked quarantined subreddits to the following: 'theredpill','cringeanarchy','braincels','subforwhitepeopleonly','americanjewishpower','cringechaos','blackfathers','4chan','accidentalnudity','bixnood','cringeanarchy','european','holocaust','ice_poseidon2','picsofdeadkids','rapefugees','starlets','theredpill','truecels','whitebeauty','youdontpass','tha_pit_pit','thinspocommunity','niggas','americanjewishpower','braincels','britishjewishpower','cringeanarchy','cringechaos','cursedx100images','cursedx3images','debatealtright','deformedbabies','edfood','fragilejewishredditor','fullcommunism','gentilesunited','holocaustfacts','i_love_niggers','ice_poseidon','identitarians','kangznsheeit','mayo_town','northwestfront','offensivememes','okbuddyanarchy','scroogeland','spacedicks','timetogo','zog' |
| Feature | 2019-04-14 | Added new endpoint: /visualize (in Alpha) |
| Status | 2019-04-15 | February comments are now available (daily files) here: https://files.pushshift.io/reddit/comments/daily/ -- Monthly file now available (as .zst) |
| bug fix | 2019-04-23 | The max_result_window size was not set correctly after I reindex a lot of data. This caused issued with removeddit choking on older submissions since they request 20k comments at a time but ES had a max of only 10k. |
| Backend | 2019-05-14 | Added an additional ingest account to increase the number of comments and submissions that can be ingested. This is mainly to deal with periods of high spam. |
| Backend | 2019-05-21 | Enhanced the comment score update script to use multiple dev apps to handle the increased load of comments. This will also accelerate getting data when the system falls behind for whatever reason. |
Author
Account Strength
100%
Account Age
11 years
Verified Email
No
Verified Flair
No
Total Karma
143,730
Link Karma
34,810
Comment Karma
108,242
Profile updated: 2 days ago
Posts updated: 6 months ago
Subreddit
Post Details
Location
We try to extract some basic information from the post title. This is not
always successful or accurate, please use your best judgement and compare
these values to the post title and body for confirmation.
- Posted
- 5 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/pushshift/c...