08-30-2007
Hi ilan,
Thanks for taking this up ... I have the first piece figured out ... I can generate a random number using a small perl script that I downloaded of the net ... but I have a problem with the second part ... I'll try to describe it better.
I have 50 million records with 2 columns. Both the columns are present in all the records.
Step1: Generate a random value (this is the part i figured out above)
Step2: Locate a random record among the 50 million
Step3: Replace the value in the second column with the value generated in step 1.
Step4: Go back to Step1, generate a new value, look for another random record, replace it with this value and so on for about a million times.
I want to be able to do this in place since everytime I replace a record using awk, it gives the whole 50 million inclusive of that change as the output and i have redirect the output to another file, rename it to the original and start over again for the next iteration.
What I need is a way to edit the file in place in a loop identifying random records and changing the second column a million times.
The high level requirement is:
Given a file of 50 million records, I have to generate a file that has 50 million records but has 1 million records whose second column varies from that of the first file. Maybe there is an easier way to do this ... But I am stumped right now ....
Thanks,
V
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
How to edit large file using vi where you can't increase /usr/var/tmp anymore? (3 Replies)
Discussion started by: nazri
3 Replies
2. Shell Programming and Scripting
I am trying to edit a file that has 33k+ records. In this file I need to edit each record that has a 'Y' in the 107th position and change the 10 fields before the 'Y' to blanks. Not all records have a 'Y' in the 107th field.
ex:
... (8 Replies)
Discussion started by: jxh461
8 Replies
3. Shell Programming and Scripting
hi All,
Plz let me know how to edit a file with 2000000 records.
each record contains with 40 field seperated by |.
i want modify 455487 record, but i am uable to edit this large file using vi editor in unix.
plz let me know how to modify this file.
Thanks in advance.
-Bali Reddy (3 Replies)
Discussion started by: balireddy_77
3 Replies
4. Shell Programming and Scripting
Hi,
I need to make a script to edit a file. File is a large file in below format
Version: 2008120101
;$INCLUDE ./abc/xyz/Delhi
;$INCLUDE ./abc/xyz/London
$INCLUDE ./abc/xyz/New York
First line in the file is version number which is in year,month,date and serial number format. Each... (5 Replies)
Discussion started by: makkar4u
5 Replies
5. Shell Programming and Scripting
Hello,
I am attempting to create a command that I can eventually put into a loop so I can edit 1file on many servers. I would like to edit the file in place with sed -i. If not I will take any suggestions on how to use a temp file. I need to remove a email address from the configuration file... (4 Replies)
Discussion started by: abacus
4 Replies
6. Solaris
Dear Gurus,
I need you to advice or suggestion about the best solution to copy data around 200-300G from serverA(location A) to serverB(location B). Normally, I will share folder and then copy but it takes too long time(about 2 days).
Do you have any suggestion or which way should be... (9 Replies)
Discussion started by: unitipon
9 Replies
7. Shell Programming and Scripting
Hello, I am using tcsh on AIX.
I would like to write a script that does the following:
1. given an inode, how do I find exactly the name of the file?
I know I could do this using ls -i | grep <inode>
but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
Discussion started by: lastZenMaster
1 Replies
8. Shell Programming and Scripting
Hi,
I lack the utter fundamentals on how to craft an awk script.
I have hundreds of text files that were mangled by .doc format so all the lines are broken up so I need to join all of the lines of text into a single line. Normally I use vim command "ggVGJ" to join all lines but with so many... (3 Replies)
Discussion started by: n00ti
3 Replies
9. Shell Programming and Scripting
Whenever I am trying to edit a file in unix with vi editor, I am getting the following error:
<data>
:Tmp file too large
Is there any way that I can edit the file other than vi. Any help is really appreciated.
Thanks (10 Replies)
Discussion started by: bobby1015
10 Replies
10. Shell Programming and Scripting
Hi All,
Daily I am getting the updated file.
I have to search for this file in all directories and sub directories.
If the file existed in a particular directory then move this updated file to that particular directory.
If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies
LEARN ABOUT NETBSD
shuffle
SHUFFLE(1) BSD General Commands Manual SHUFFLE(1)
NAME
shuffle -- print a random permutation of the command line arguments
SYNOPSIS
shuffle [-0] [-f filename ...] [-n number] [-p number] [arg] [...]
DESCRIPTION
The shuffle program prints a random permutation (or ``shuffle'') of its command line arguments. This can be useful in shell scripts for
selecting a random order in which to do a set of tasks, view a set of files, etc.
If the -f option is given, the data is taken from that files' contents or if the filename is - ``stdin''.
If the -n option is given, its argument is treated as a number, and the program prints a random permutation of the numbers greater than or
equal to 0 and less than the argument.
If the -p option is given, its argument is treated as a number, and the program prints that number of randomly selected lines or arguments in
a random order.
The -0 option changes the field separator character from
to , so that the output is suitable to be sent to xargs(1) (to handle filenames
with whitespace in them).
EXAMPLES
$ shuffle a b c d
c
b
d
a
$ shuffle -p 1 a b c d
d
$ shuffle -n 4 -p 2
0
3
SEE ALSO
jot(1), random(6)
HISTORY
The shuffle program first appeared in NetBSD 1.4.
AUTHORS
Written by Perry E. Metzger <perry@piermont.com>.
BSD
February 18, 2009 BSD