Cfrtjryk

I have been working on a project in python which takes content from web links and find some important words from content of that page.
I used regular expression to do so.
But it takes huge time to get the results.

How it works:

1. It makes request to given url


2. take whole html content and data from external js files(within src attribute of tag)


3. Save those things to list. (name it as file, used in re later)


4. and then perform searching of important data using regular expression.

Here is the sample:

List:

list = ['secret', 'secret_key', 'token', 'secret_token', 'auth_token','access_token','username','password','aws_access_key_id','aws_secret_access_key', 'secretkey']

Regular Expression:

for item in seclst:

    try:

        secregex = r'(["']?[w-]*' + item + '[w-]*[s]*["']?[s]*[:=>]{1,2}[s]*["'](.*?)["'])'

        matches = re.finditer(secregex, file, re.MULTILINE | re.IGNORECASE )

        for matchNum, match in enumerate(matches):

            if len(match.group(2)) > 0:

                secretList.add(match.group())

    except:

        pass

There are some other functions too.

Execution time and explanation:

1. When I use 'https://www.facebook.com'(without cookies) it takes approximately 41 seconds (including doing other functionalities)


2. When I use 'https://www.facebook.com'(with cookies) it takes approximately 5 to 6 min (including doing other functionalities)

How can I optimize it?