Cfrtjryk

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?

from bs4 import BeautifulSoup

import requests

import csv



source = requests.get("https://www.youtube.com/feed/trending").text

soup = BeautifulSoup(source, 'lxml')



csv_file = open('YouTube Trending Titles on 12-30-18.csv','w')

csv_writer = csv.writer(csv_file)

csv_writer.writerow(['Title', 'Description'])



for content in soup.find_all('div', class_= "yt-lockup-content"):

    try:

        title = content.h3.a.text

        print(title)



        description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text

        print(description)



    except Exception as e:

        description = None



    print('n')

    csv_writer.writerow([title, description])



csv_file.close()

Why web-scrape, when you can get the data properly through the YouTube Data API, requesting the mostpopular list of videos? If you make a GET request to https://www.googleapis.com/youtube/v3/videos?key=…&part=snippet&chart=mostpopular, you will get the same information in a documented JSON format.

Using the Python client, the code looks like:

import csv

import googleapiclient.discovery



def most_popular(yt, **kwargs):

    popular = yt.videos().list(chart='mostPopular', part='snippet', **kwargs).execute()

    for video in popular['items']:

        yield video['snippet']



yt = googleapiclient.discovery.build('youtube', 'v3', developerKey=…)

with open('YouTube Trending Titles on 12-30-18.csv', 'w') as f:

    csv_writer = csv.writer(f)

    csv_writer.writerow(['Title', 'Description'])

    csv_writer.writerows(

        [snip['title'], snip['description']]

        for snip in most_popular(yt, maxResults=20, regionCode=…)

    )

I've also restructured the code so that all of the CSV-writing code appears together, an inside a with open(…) as f: … block.

Context manager

You open a file at the beginning of the program and close it explicitly at the end.

Python provides a nice way to allocate and release resources (such as files) easily: they are called Context managers. They give you the guarantee that the cleanup is performed at the end even in case of exception.

In your case, you could write:

with open('YouTube Trending Titles on 12-30-18.csv','w') as file:

    ....

Exception

All exceptions are caught by except Exception as e. It may look like a good idea at first but this can lead to various issues:

• it's hard to know what types of error are actually expected here


• most errors are better not caught (except for special situations). For instance, if you write a typo, you'll end up with an ignored NameError or AttributeError and debugging will be more painful than it should be.

Also, from the content of the except block, it looks like you are only expecting the logic about description to fail. If so, it would be clearer to put in the try (...) except the smallest amount of code.

For instance:

title = content.h3.a.text

try:

    description = content.find('div', class_="yt-lockup-description yt-ui-ellipsis yt-ui-ellipsis-2").text

except Exception as e:

    description = None

print(title)

print(description)

print('n')

Proper solution

Google usually offers API to retrieve things such like trending videos. I haven't found it but I'll let you try to find something that works properly. Google is your friend...