This post has been de-listed
It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.
Hey guys, sorry in advance if this question is a little obvious, I'm new to programming and Python. I am trying to scrape data off a website with over 30,000 products. I followed a YouTube video that broke it down and was able to successfully print 250 item names in the terminal.
Once I started adding the variants and imagesrc code and pandas to convert the data to a CSV file, it gives me an error of invalid syntax.
I was also wondering if there is anyway to download all 30,000 products in one go or would I have to modify '&page=1' to '&page=2', '&page=3' and so on.
I pretty much just need the product name, tags [1] and [3], image(s) if there is one and price. Looking for some guidance on how to do this.
I have attached the code below that is giving me the invalid syntax error. Here is a screenshot of how it looks as well.
import requestsimport jsonimport pandas as pdurl = 'https://plus.mvrwholesale.com/products.json?limit=250&page=1'r = requests.get(url)data = r.json()product_list = []for item in data ['products']:title = item ['title']handle = item['handle']product_type = item['product_type']for variant in item['variants']:price = variant ['price']sku = variant['sku']available = variant ['available']product = {'title': title,'handle': handle,'created': created,'product_type': product_type,'price' : price,'sku': sku,'available' : available,
}product_list.append(product)
df = pd.DataFrame (product_list)df.to_csv('testrun.csv')print('saved to file.')
Subreddit
Post Details
- Posted
- 2 years ago
- Reddit URL
- View post on reddit.com
- External URL
- reddit.com/r/pythonhelp/...