Cfrtjryk

In this SO question the OP is unable to scrape a table from a dynamically loaded website. In monitoring the web traffic, via Chrome dev tools, I found that there is an API request made that returns a JSON string with the required info.

The following is the answer I wrote to extract the columns of interest from the API response.

Guide to columns of interest:

1. 
   Course Title = title
   


2. 
   Trainer = name (within trainers)


3. 
   Rating = rating
   


4. 
   Vendor = name (within vendors)


5. 
   IT Path = path_label (within paths)


6. 
   Skill Level = display (within difficulty)


7. 
   Course URL = concatenation of base with seoslug
   

The Vendors field has missing items hence my use of an if statement in the assigment to vendors. I am not sure what the usual placeholder value is for missing string values in Python.

I use repeated list comprehensions in loops over the JSON object data; where data = response.json()

I couldn't think of a way to remove these repeated loops and still have legible code.

I generate a dataframe by joining the lists in a dictionary and then converting with pandas.

I welcome any and all feedback please.

JSON response:

Example JSON dictionary within response. The response has a collection of such dictionaries.

Python 3

import requests

import pandas as pd





def main():

    base = 'https://www.cbtnuggets.com/it-training/'

    response  = requests.get('https://api.cbtnuggets.com/site-gateway/v1/all/courses/for/search?archive=false')



    data = response.json()



    titles = [item['title'] for item in data]

    trainers = [item['trainers'][0]['name'] for item in data]

    ratings = [item['rating'] for item in data]

    vendors = [item['vendors'][0]['display'] if len(item['vendors']) != 0 else 'N/A' for item in data]

    paths = [item['paths'][0]['path_label'] for item in data]

    skillLevel = [item['difficulty']['display'] for item in data]

    links = [base + item['seoslug'] for item in data]



    df=  pd.DataFrame(

      {'Course Title': titles,

       'Trainer': trainers,

       'Rating': ratings,

       'Vendor': vendors,

       'IT Path': paths,

       'Skill Level': skillLevel,

       'Course URL': links

      })



    #print(df)

    df.to_csv(r'C:UsersUserDesktopData.csv', sep=',', encoding='utf-8',index = False )



if __name__ == "__main__":



    main()

I couldn't think of a way to remove these repeated loops and still have legible code.

There is a way:

titles, trainers, ratings, vendors, paths, skillLevel, links = zip(*((

    item['title'],

    item['trainers'][0]['name'],

    item['rating'],

    item['vendors'][0]['display'],

    item['paths'][0]['path_label'],

    item['difficulty']['display'],

    base + item['seoslug']

) for item in data))

I haven't tested this, so you should.

The only major thing I would change is that:

if len(item['vendors']) != 0

Is the same as:

if item['vendors']

Because an empty list will return back as False. If you want to try it out:

a = 

bool(a)      # False

b = [1,2,3]

bool(b)      # True

I would also be careful with what you have because those dictionaries that you are converting might have more than one value, in which case you would miss them. This is the line that I am referring to:

    paths = [item['paths'][0]['path_label'] for item in data]

Instead of splitting the data into a separate variables by column, you could convert each JSON object into a flat dictionary using a function similar to this

def course_dict(item):

    return {'Course Title': item['title'],

            'Vendor': item['vendors'][0]['display'] if item['vendors'] else None,

            # and so on

            }

and construct the dataframe using

data = response.json()

df = pd.DataFrame([course_dict(item) for item in data])

Keeping related data together makes the code easier to follow. Also, since your final output is a csv file, you could skip the dataframe and use csv.DictWriter instead.

I am not sure what the usual placeholder value is for missing string values in Python.

None is the usual placeholder for missing values of any type.