UNIX for Dummies Questions & Answers

Lets say I have a file abc.txt and it has about 35 million rows. I would like to take a sample of 100 random rows from that file for my testing purpose and write it to a file say test.txt.

How do I do this operation?

Thanks,
Sashank

If you need it to be truly random, you'll need a suitable random number generator.

Once you have 100 random (or predefined if that's all you need) numbers, normalise them against the number of lines in your file (I'd suggest you select 100 numbers between 0 and 1, then multiply that by the lines in the file). That will give you a list of 100 lines numbers.

Code:

#!/bin/sh
# make LINES contain a space seperated list of your line numbers
INPUT="abc.txt"
OUTPUT="test.txt"
rm -f $OUTPUT
for line in $LINES
do
  head -$line $INPUT | tail -1 >> $OUTPUT
done

It's clumsy and won't be very quick (especially for lines further into the file) but it'll do the job.

A better solution would be to seek into the file the correct distance and just dd out the required data.

Why not use sed in place of the head/tail?

sed -n "$line p" filename >> $OUTOUT

Didn't know sed could do that... Handy I wonder which one uses more cycles as they both would have to parse and reparse the file... I suspect yours will be quicker

This was something I picked up from these forums a couple of days back. Didn't know I would be passing on this "knowledge" to others so soon.

About the cycles: yes, I think so.

Much faster use of sed...

Code:

 sed -n ${line}'{p;q;}' filename

Upon reaching the desired line, sed will print it and then immediately exit. It will not continue uselessly reading the rest of the file.

You can combine all of them into a single sed script, which will be a lot quicker.

Say, given a list of numbers in increasing order in file1, you can

Code:

sed -e 's/$/p/' file1 | sed  -n -f - bigfile >samples

to read the big file just once, and print the selected line numbers.

(Not all sed implementations understand "-f -" I have been stymied to learn; you need a temporary file then, obviously.)

If you are a bit clever you can also make it quit after printing the last one, to avoid needlessly reading the big input file through to the end. Implementing that is left as an exercise for the astute reader. (-: