This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
4
Trying to scrape a website but lost
Post Body
I am using grab-site. Have very basic technical knowledge of this area but I am happy to learn. Have run into several issues so far.
- Not able to restart a session once I stop it. Didn't see any option in grab-site. Tried using --wpul-args=--database and --load-cookies but I am not sure how to use multiple arguments. Would appreciate an example. Here is what I did:
grab-site --concurrency=1 --delay=100-500 --warc-max-size=1068709120 --wpull-args="--database=ABS_PATH/wpull.db --load-cookies=ABS_PATH/cookie.txt" URL - The rest of the content is being downloaded but mp4 media download always fails with 'connection timeout'. Maybe I am not passing the cookies properly or maybe the download requires some kind of setup that grab-site doesn't do? e.g.
ERROR Fetching ‘https://149.11.89.180/etv/content/D5KuA.mp4.m3u8?e=1700914018&md5=ISTqhJdrsxsXUjT3QmVuqA’encountered an error: Connect timed out.
Any help is appreciated.
Author
Account Strength
100%
Account Age
5 years
Verified Email
Yes
Verified Flair
No
Total Karma
2,183
Link Karma
443
Comment Karma
1,740
Profile updated: 2 days ago
Posts updated: 2 weeks ago
Subreddit
Post Details
We try to extract some basic information from the post title. This is not
always successful or accurate, please use your best judgement and compare
these values to the post title and body for confirmation.
- Posted
- 1 year ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/Archivists/...