UNIX for Dummies Questions & Answers

Swings and roundabouts. For data amounting to a million lines my suggestion is slightly slower for say 20,000 small files but faster for a small number of large files because "cat" is more efficient than "wc" at reading from disc. My method gives much less of a cpu hit. In the real world I use both constructs.

Code:

This is interesting.

time wc -l bigfile
5487935 bigfile

real       12.1
user       10.7
sys         1.3

time cat bigfile|wc -l

real       12.0
user        0.1
sys         2.5
5487935

you only had one run of each. do more runs and find out.

find . -type f | xargs cat |wc -l[COLOR="#738fbf"]

replacing . with the directory name ;-)

(not sure what good timing wc is. But interesting. I'm guessing loading two programs (wc and cat) into memory is slower that loading just one? I'd love it if Microsoft would bear that in mind! (oops, probably violated a rule there... sorry)

Hmm. "scottn" idea breaks if any filenames contain space characters.


I ran my test a few times first to eliminate o/s first-time buffering before posting those figures. Results were still interesting.

The "pludi" solution is very good and exhibits lateral thought, but on this occasion the UUOC argument is arguable because on my system "wc" is less efficent at reading files from disc than "cat".

BTW. I can produce the required output by using only "find" and shell commands but it proved to be horrendously slow to read large volumes of data with a shell read.



There is life in the old cat yet!

Sorry, I got a bit confused as to what thread I was writing in when I said that. I meant no disrespect. It is interesting.

You're right, but Unix filenames don't "normally" have spaces.

Time: 8 - 10 seconds (on some FS full of rubbish on my system):

Code:

find . -type f | xargs -I{} cat "{}" | wc -l

Time: 2 - 3 seconds (but doesn't handle some files):

Code:

find . -type f | xargs cat | wc -l

Ergo: awk rocks (and cat is cool)

Did you run each example more than once, or are the examples you posted exactly what you did test?

After the first run, depending on your filesystem and OS, there's a good chance that a lot of the data is going to be cached and not read from disk.

I ran them lots of times.... but I was running Linux in a VM on Windows. And who knows what that get's up to while you're not looking!

The point is, it seems, trusting that your filenames don't have spaces - and you don't have to check for them, it's quicker than if you do need to check.

In any case, the awk solution was nicer.