Cfrtjryk

I want to read webpage list and check if some of them have updated. Is it better to use wget or curl and how should I do that?

Webpage list is in a simple text file. If the contents of a webpage is the same it will not print anything. If contents changed from the last time that the script ran, then it will type(stdout) the webpage address.

#!/bin/sh



i=1

while IFS= read -r url; do

    file="data-$i.out"



    curl -o "$file.new" "$url"



    if ! cmp -s "$file" "$file.new"

    then

        printf '%sn' "$url"

    fi



    mv -f "$file.new" "$file"



    i=$(( i + 1 ))

done <url-list.txt

This would read the URLs from url-list.txt, line by line, and use curl to fetch each, saving the output in a file called data-N.out.new where N is an integer (the URL ordinal number in the file).

If there is no old data-N.out file, or if this file differs from data-N.out.new, then the URL is printed to standard output.

The fetched data file is then renamed for when you run the script again.

The first time you run the script, all URLs will be outputted as they have never been seen before.

Reordering the URLs, or adding new URLs at the top would make the URLs be flagged as changed as the contents of the corresponding data file has changed. You could fix this by using e.g. the base64-encoded URL as part of the output filename instead of $i.

Whether you use curl or wget or some other Web client is essentially unimportant.