​

Hi so im trying to scrape the league of legends website for its champion names using python and bs4. There is probably a website thats easier to scrape with the same information but I was wondering in this situation how would i scrape it. I am pretty sure they load it in from database or from some block tags so its not actually inside the soup. I cant find the the div tags with the relevant information.

Heres the website: https://universe.leagueoflegends.com/en_US/champions/

Would I have to use a scrapper module like selenium to actually go to the website and read the <ul> tags that way?

Here is my code:

​

import requests

import bs4 as bs

list_of_champs = []

url_for_list = 'https://universe.leagueoflegends.com/en_US/champions/'

def parse():

request = requests.get(url_for_list)

soup = bs.BeautifulSoup(request.text,'xml')

print(soup.prettify())

for x in soup.find_all('div'):

print(x)

if __name__ == '__main__':

parse()