UNIX for Beginners Questions & Answers

That's not how fifo's work.

Code:

mkfifo /tmp/jobqueue

# Need to put this in the background, since it will wait until both
# ends of the fifo are open
( wget -q -r -l1 -O /tmp/jobqueue URL/ & )

< /tmp/jobqueue grep -hrio "\b[a-z0-9.-]\+@[a-z0-9.-]\+\.[a-z]\{2,4\}\+\b" |
    sort -u |
    sed '0~5 a\\' > 7.txt

wait

rm -f /tmp/jobqueue

That's interesting, thanks. Probably getting closer?

Code:

>     sed '0~5 a\\' > 7.txt
-k or -r can be used together with -O only if outputting to a regular file.

That's the clue we needed - the reason it doesn't work when you put the same command in a pipe chain.

It doesn't work simply because wget refuses to output to anything but a regular file. Pipes, named or otherwise, are no good. For -k, this is because it has to edit the file after it downloads it. For -p, this is because it opens the output file more than once. Both of which would indeed do awful things to a pipe.

So, not a lot you can do. Sorry.

Thanks for explanation; you'll see I did post that error message a few posts ago.

So I'll have to make a new directory every time I run this sequence...pretty inefficient. Anyhow if you or anyone else gets another idea on this please update. Thanks.

I don't see why using a folder makes it inefficient.. What exactly are you trying to avoid?

Making, and cd'ing into all those nested directories is inefficient, when I might have all the URL folders in 1 directory.

If you don't want files in folders, why are you downloading recursively?

If you want multiple files, why are you downloading recursively, when you'll have no order over the order they're retrieved?

What exactly are you trying to do, that wget is not doing? What is the grep actually for? What sort of data are you retrieving?

Perhaps explain, from the beginning, what you are actually trying to do? I think we may have taken a wrong turn somewhere.