Coming soon - Get a detailed view of why an account is flagged as spam!
view details
5
Trying to get a scraper working. Program runs but does not output to log.txt
Post Body

Not sure what Is going wrong the program runs but does not seem to be outputting anything.

import pycurl
import json
import time
from io import BytesIO

i = 0
bnums = []

while True :
print(str(i))
buffer = BytesIO()
c = pycurl.Curl()
c.setopt(c.WRITEDATA, buffer)
c.setopt(c.ENCODING, 'gzip,deflate')
c.setopt(pycurl.HTTPHEADER, ['Pragma: no-cache',
    'Accept-Language: en-US,en;q=0.8,it;q=0.6',
    'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36',
    'Accept: text/javascript, text/html, application/xml, text/xml, */*',
    'X-Prototype-Version: 1.7.3',
    'X-Requested-With: XMLHttpRequest',
    'Cookie: PHPSESSID=brfmu8t0mopjk7l96gooerm9b3; bzc=8jdes06rltfc; pwr_new_tooltip=true; bzu=r8desr59d4ds; pwr_selc_auctid=false',
    'Connection: keep-alive',
    'If-Modified-Since: Thu, 1 Jan 1970 00:00:00 GMT',
    'Referer: http://www.beezid.com/morphic-men-s-m34-series-watch-white-dial',
    'Cache-Control: no-cache'
    ])
c.setopt(c.URL, 'http://www.beezid.com/auctions/updater/id:1157971/bid:' str(i))
c.perform()
c.close()
js = buffer.getvalue()
try:
    jl = json.loads(js)
    if len(jl['bids']) > 0:
        for bid in jl['bids']:
            if bid['id'] > i:
                i = bid['id']
            if any(str(bid['id']) in s for s in bnums):
                pass
            else:
                bnums.append(str(bid['id']))
                with open("log.txt", "a") as myfile:
                    myfile.write(json.dumps(bid))
                    myfile.write("\n")
except:
    pass
time.sleep(.5)

Author
Account Strength
100%
Account Age
9 years
Verified Email
Yes
Verified Flair
No
Total Karma
158,283
Link Karma
22,708
Comment Karma
135,439
Profile updated: 7 hours ago
Posts updated: 1 month ago

Subreddit

Post Details

We try to extract some basic information from the post title. This is not always successful or accurate, please use your best judgement and compare these values to the post title and body for confirmation.
Posted
8 years ago