This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
After hours of successfully yet fairly slowly getting requests from subreddit "selfie" this piece of code:
if resp_.to_string().contains("Too Many"){
println!("2many rqstz");
'rqst:loop{
println!("4");
thread::sleep(time::Duration::from_secs(1));
resp_ = client.get("https://api.pushshift.io/reddit/search/submission/")
.headers(construct_headers())
.send()
.await? //here is the problem. no error, and no resolution
.text()
.await?
;
if resp_.to_string().contains("Too Many"){
println!("2mny");
continue
} else {
break 'rqst
};
};
};
gets stuck at the end of the subreddit. The response is awaited forever, nothing changes, nothing happens, there's nothing in response.
As you can see in the logic of the code - if there's a rate-limit, the code makes it so that less than 60 requests are sent per minute and only proceeds when the response is valid. Here it's stuck at just the first await?
It doesn't give out any error, obviously, it's just awaited forever.
Output after hours of success is that major fail:
next iter ]"selfie"[ -> ["1644793985"]
LAST:"1644785041"
(...)
2many rqstz
4
And it's forever stuck here. "LAST" output means - the last "created_utc" timestamp from previous/current json response - the current json response is obtained with the first timestamp, like so: "(...)&?before=1644793985". If there is no timestamp there, this value can't change, and yet it does, so the current/previous response is valid. During scraping, it's also sometimes getting obnoxiously long await times for response, much more than my rate limit.
Subreddit
Post Details
- Posted
- 2 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/pushshift/c...