Web scrapping a javascript oriented website

This post has been de-listed

It is no longer included in search results and normal feeds (front page, hot posts, subreddit posts, etc). It remains visible only via the author's post history.

Post Body

I'm trying to extract data (competition location, date, etc) from this page : https://components.ifsc-climbing.org/calendar/. The data is inside multiple div of class competition.

So far, I managed to get the html of the page using requests or Selenium :

import requests
from selenium import webdriver

browser = webdriver.Chrome('/usr/local/bin/chromedriver')
url = 'https://components.ifsc-climbing.org/calendar/'
browser.get(url)

html = browser.page_source
browser.quit()

print(html)

The problem is that I think the data I'm looking for is added with a javascript script (disabling javascript from the page on my browser breaks it) and I can't find a way to get it to work. I've tried to execute the scripts using requests_html render function:

import requests
from requests_html import HTMLSession

session = HTMLSession()
resp = session.get("https://components.ifsc-climbing.org/calendar/")
resp.html.render()

print(resp.html.html)

But both scripts give the same result without the information I'm looking for:

<html><head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

        <!-- Bootstrap CSS -->
        <link rel="stylesheet" href="../utils/assets/css/bootstrap.min.css">
        <link rel="stylesheet" type="text/css" href="../utils/assets/css/jquery.dataTables.min.css">

        <!-- Custom CSS -->
        <link rel="stylesheet" href="./assets/css/custom.css">
    </head>

    <body class="oswaldFont">
        <div class="calContainer">
            <div class="ifsc_calendar">
                <h1 id="cal_year"></h1>
                <div class="filter_select">
                    <select class="customSelect" id="yearSelect"></select>
                    <select class="customSelect" id="leaguesSelect"></select>
                </div>
                <div class="events"></div>
            </div>
        </div>


        <!-- JS -->
        <script src="../utils/assets/js/jquery-3.4.1.min.js"></script>
        <script type="text/javascript" charset="utf8" src="../utils/assets/js/jquery.dataTables.min.js"></script>

        <!-- Custom JS -->
        <script src="./assets/js/calendar.js"></script>
        <script src="../utils/environment.js"></script>

</body></html>

Any hints on how to get this working?

Author

Account Strength

100%

Account Age

7 years

Verified Email

Yes

Verified Flair

Total Karma

5,457

Link Karma

1,172

Comment Karma

4,204

Profile updated: 5 days ago

Posts updated: 4 months ago

BenBenBenz

Subreddit

r/learnprogramming

Post Details

We try to extract some basic information from the post title. This is not always successful or accurate, please use your best judgement and compare these values to the post title and body for confirmation.

Posted: 3 years ago
Reddit URL: View post on reddit.com
External URL: reddit.com/r/learnprogra...