How to read and write a random row from a file?

sashankkrk · #1 (**permalink**) 05-08-2008

Lets say I have a file abc.txt and it has about 35 million rows. I would like to take a sample of 100 random rows from that file for my testing purpose and write it to a file say test.txt.

How do I do this operation?

Thanks,
Sashank

Smiling Dragon · #2 (**permalink**) 05-12-2008

If you need it to be truly random, you'll need a suitable random number generator.

Once you have 100 random (or predefined if that's all you need) numbers, normalise them against the number of lines in your file (I'd suggest you select 100 numbers between 0 and 1, then multiply that by the lines in the file). That will give you a list of 100 lines numbers.

Code:

#!/bin/sh
# make LINES contain a space seperated list of your line numbers
INPUT="abc.txt"
OUTPUT="test.txt"
rm -f $OUTPUT
for line in $LINES
do
  head -$line $INPUT | tail -1 >> $OUTPUT
done

It's clumsy and won't be very quick (especially for lines further into the file) but it'll do the job.

A better solution would be to seek into the file the correct distance and just dd out the required data.

frozentin · #3 (**permalink**) 05-13-2008

Quote:

Originally Posted by Smiling Dragon

If you need it to be truly random, you'll need a suitable random number generator.

Once you have 100 random (or predefined if that's all you need) numbers, normalise them against the number of lines in your file (I'd suggest you select 100 numbers between 0 and 1, then multiply that by the lines in the file). That will give you a list of 100 lines numbers.

Code:

#!/bin/sh
# make LINES contain a space seperated list of your line numbers
INPUT="abc.txt"
OUTPUT="test.txt"
rm -f $OUTPUT
for line in $LINES
do
  head -$line $INPUT | tail -1 >> $OUTPUT
done

It's clumsy and won't be very quick (especially for lines further into the file) but it'll do the job.

A better solution would be to seek into the file the correct distance and just dd out the required data.

Why not use sed in place of the head/tail?

sed -n "$line p" filename >> $OUTOUT

Smiling Dragon · #4 (**permalink**) 05-13-2008

Quote:

Originally Posted by frozentin

Why not use sed in place of the head/tail?
sed -n "$line p" filename >> $OUTOUT

Didn't know sed could do that... Handy

I wonder which one uses more cycles as they both would have to parse and reparse the file... I suspect yours will be quicker

frozentin · #5 (**permalink**) 05-13-2008

This was something I picked up from these forums a couple of days back. Didn't know I would be passing on this "knowledge" to others so soon.

About the cycles: yes, I think so.

Perderabo · #6 (**permalink**) 05-13-2008

Much faster use of sed...

Code:

 sed -n ${line}'{p;q;}' filename

Upon reaching the desired line, sed will print it and then immediately exit. It will not continue uselessly reading the rest of the file.

era · #7 (**permalink**) 05-13-2008

You can combine all of them into a single sed script, which will be a lot quicker.

Say, given a list of numbers in increasing order in file1, you can

Code:

sed -e 's/$/p/' file1 | sed  -n -f - bigfile >samples

to read the big file just once, and print the selected line numbers.

(Not all sed implementations understand "-f -" I have been stymied to learn; you need a temporary file then, obviously.)

If you are a bit clever you can also make it quit after printing the last one, to avoid needlessly reading the big input file through to the end. Implementing that is left as an exercise for the astute reader. (-:

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
read and write from a file	rinku	Shell Programming and Scripting	2	01-11-2008 01:22 AM
How to read and write directory or file contents in c++ ?	namrata5	High Level Programming	3	09-28-2007 03:58 PM
sed to read and write to very same file	435 Gavea	Shell Programming and Scripting	5	06-29-2006 11:04 PM
popening for read and write	szzz	High Level Programming	1	11-18-2003 12:05 PM
read, write & STDOUT_FILENO....	M3xican	High Level Programming	2	07-17-2002 04:41 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by