Home:ALL Converter>My code randomly brakes on extracting data from websites (python)

My code randomly brakes on extracting data from websites (python)

Ask Time:2021-09-22T05:25:59         Author:Kattamaran

Json Formatter

My code results in an error when scraping the url. I only post the problematic snipet. If this code gets run a couple of times it will produce this error. I have about 20 urls with product pages like this one and it always fails at random points. Sometimes it can read a url fine, other times it messes up. I find it strange, that it only happens to the "preis" variable. I tryed to add delays at several stages. Up to 5 seconds, but that does not seem to solve my problem.

File "C:\Users\kibit\PycharmProjects\scrapingRequest-html\Test.py", line 23, in <module>
    preis = article.find('div.drzWO', first=True).text
AttributeError: 'NoneType' object has no attribute 'text'


from requests_html import HTMLSession
import csv
import time

#csv_file = open('beoscrape.csv', 'w')
#csv_writer = csv.writer(csv_file)
#csv_writer.writerow(['Produkt', 'Retailer', 'Preis'])

session = HTMLSession()

r = session.get(url)

produktscan = r.html.find('title')
produkt = produktscan[0].text
produkt = produkt.replace(' | Google Shopping', '')

for article in r.html.find('tr.sh-osd__offer-row'):

    retailer = article.find('div.kPMwsc', first=True).full_text
    retailer = retailer.replace('Wird in einem neuen Fenster geöffnet', '')
    preis = article.find('div.drzWO', first=True).text


#    csv_writer.writerow([produkt, retailer, preis])


Author:Kattamaran,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/69275731/my-code-randomly-brakes-on-extracting-data-from-websites-python