Home:ALL Converter>My code randomly brakes on extracting data from websites (python)

My code randomly brakes on extracting data from websites (python)

Ask Time:2021-09-22T05:25:59         Author:Kattamaran

Json Formatter

My code results in an error when scraping the url. I only post the problematic snipet. If this code gets run a couple of times it will produce this error. I have about 20 urls with product pages like this one and it always fails at random points. Sometimes it can read a url fine, other times it messes up. I find it strange, that it only happens to the "preis" variable. I tryed to add delays at several stages. Up to 5 seconds, but that does not seem to solve my problem.

File "C:\Users\kibit\PycharmProjects\scrapingRequest-html\Test.py", line 23, in <module>
    preis = article.find('div.drzWO', first=True).text
AttributeError: 'NoneType' object has no attribute 'text'

Code:

from requests_html import HTMLSession
import csv
import time

#csv_file = open('beoscrape.csv', 'w')
#csv_writer = csv.writer(csv_file)
#csv_writer.writerow(['Produkt', 'Retailer', 'Preis'])

session = HTMLSession()
url='https://www.google.com/shopping/product/16302210309954970571/offers?q=OLED+65+G19+LA&newwindow=1&prds=cid:16302210309954970571,eto:856723695612861771_0,scoring:tp'

r = session.get(url)
r.html.render()

produktscan = r.html.find('title')
produkt = produktscan[0].text
produkt = produkt.replace(' | Google Shopping', '')

for article in r.html.find('tr.sh-osd__offer-row'):

    retailer = article.find('div.kPMwsc', first=True).full_text
    retailer = retailer.replace('Wird in einem neuen Fenster geöffnet', '')
    preis = article.find('div.drzWO', first=True).text

    print(produkt)
    print(retailer)
    print(preis)

#    csv_writer.writerow([produkt, retailer, preis])

#csv_file.close()

Author:Kattamaran,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/69275731/my-code-randomly-brakes-on-extracting-data-from-websites-python
yy