Cfrtjryk

This issue is currently driving me up the wall.

It just does not work as it should.

I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other location of the HTML source file to get rid of the internal (hex) filename, looking like this:

http://whatever.site/data/samples/hexfilename1.mp3 12345.mp3

http://whatever.site/data/samples/hexfilename2.mp3 12346.mp3

http://whatever.site/data/samples/hexfilename3.mp3 12347.mp3

http://whatever.site/data/samples/hexfilename4.mp3 12348.mp3 

http://whatever.site/data/samples/hexfilename5.mp3 12349.mp3

As I only need the first part on each line, I've tried awk or alternatively cut to strip the rest, but on the fly:

$ wget -nc -i $(cut -f1 '-d ' inp)

respectively

$ wget -nc -i $(awk 'print $1' inp)

But it will download all the mp3 files, then grind for a short while, and something very strange will happen:

--2014-09-01 14:27:25--  http://whatever.site/data/samples/ID3%04

Ugh. It is exactly what you're thinking it is: indeed the first bytes of the binary mp3 file that wget is trying to download, after it is finished downloading the regular ones (and supposed to terminate). But why does it happen?
If I go the clumsy way by creating a inp2 temporary file for wget and using it with the -i parameter, it works:

$ cat inp | awk '{print $1}' > inp2

Why is there so much difference when inp gets modified on the fly and passed directly to wget?
The most interesting thing is that the on-the-fly variant won't work with either awk or cut, so neither of both tools are to blame.

The reason it didn't work is bad syntax:

wget -nc -i $(cut -f1 '-d ' inp)

...the problem is the -i switch expects either:

1. a local text file containing list of URLs


2. a remote text file containing list of URLs


3. a remote HTML file containing list of files local to it.

But the code above gives -i http://whatever.site/data/samples/hexfilename1.mp3, which isn't a text or HMTL file. man wget says:

COLUMNS=72 man wget | grep -m1 -A 22 '-i '

   -i file

   --input-file=file

       Read URLs from a local or external file.  If - is specified

       as file, URLs are read from the standard input.  (Use ./-

       to read from a file literally named -.)



       If this function is used, no URLs need be present on the

       command line.  If there are URLs both on the command line

       and in an input file, those on the command lines will be

       the first ones to be retrieved.  If --force-html is not

       specified, then file should consist of a series of URLs,

       one per line.



       However, if you specify --force-html, the document will be

       regarded as html.  In that case you may have problems with

       relative links, which you can solve either by adding "<base

       href="url">" to the documents or by specifying --base=url

       on the command line.



       If the file is an external one, the document will be

       automatically treated as html if the Content-Type matches

       text/html.  Furthermore, the file's location will be

       implicitly used as base href if none was specified.

Fixes include:

1. 
   

   Using stdin for the -i parameter as per garethTheRed's
   comment:

   
   
   

   cut -d' ' -f1 inp | wget -nc -i -
   
   

   
   


2. 
   

   Or this bash centric method, which is about one byte off from what
   was originally intended, as per syntaxerror's comment :

   
   
   

   wget -nc -i <(cut -f1 '-d ' inp)