Generating millions of record using shell script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Generating millions of record using shell script
# 1  
Old 12-11-2009
Generating millions of record using shell script

Hi All,

My requirement is like this.
I want to generate records of 1 million lines. If I say lines it means one line will contain some string or numbers like
Code:
AA,3,4,45,+223424234,Tets,Ghdj,+33434,345453434,........................ upto length lets say 41. ( 41 comma sepearted aplha numneric values ).
Now i want 5, 7 and 23 field should be unique or in some incremental order for ex.  if in first line value is 33 then next line, same fields position value should be 34 and so on. All other fields in all the lines can be same
Now for reference this script should refer one sample file and based of that it will create a file with 1 million lines. In my case following is my reference file

1,10,+91900000000,+9112323230000,,1,60,25/1/200923:10:10,3534671234500001,40591912000000001,1001001,1,1,10,3,1,1,1,0,20,+91900000000,329,ss,in_flag,digits,IC_TRUN,OG_TRUNK,NET_IND,realer_st,switch,source,plmn,1,cug_code,ff_call,corela,service3g,conf,op1,op2,op3

I want 3,4 and 21 field as as unique rest filed can be same.

Please help me out in this as am new to shell scripting.

rd-
# 2  
Old 12-11-2009
You can try something like this.

In this example, 2nd,3rd,5th fields are unique in each line.

Code:
for i in $(seq 10);
    do
        echo "a,2,c,d,5,6" | awk -v v=$i 'BEGIN {FS=OFS=","} {$2=v$2;$3=v$3;$5=v$5}{print}'
done

Output:
a,12,1c,d,15,6
a,22,2c,d,25,6
a,32,3c,d,35,6
a,42,4c,d,45,6
a,52,5c,d,55,6
a,62,6c,d,65,6
a,72,7c,d,75,6
a,82,8c,d,85,6
a,92,9c,d,95,6
a,102,10c,d,105,6

You can also use Bash "RANDOM" to make them very unique.
# 3  
Old 12-11-2009
It really worked....

Thanks a tonne, it really worked after doing some customization to your script to get my desired result.

Once again thanks for replying so quiklySmilieSmilie

My final Script is somethin like this
Code:
#!/bin/bash
> VoiceCDR.txt
echo "Recordtype:1"  >> VoiceCDR.txt

for i in $(seq 1000);
    do
        echo "2009/11/07 13:45:15,1024,+9199511000,+9198000000,,1,42,2009/11/07 13:45:15,,4059192908198221,,3,1025,0.596,1,1,1,1,,0,0,+9199511000,,CallType_Out_Int,,,919032006820,,,,,,,,19,,0,,1,,Account-1,269192908198221,,,," | awk -v v=$i 'BEGIN {FS=OFS=","} {$3=+$3+v;$4=+$4+v;$22=+$22+v}{print}' >> VoiceCDR.txt
done

perl -pi -e "s:,91:,+91:g" VoiceCDR.txt

rd-
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Is there a way to handle commas inside the data when generating a csv file from shell script?

I am extracting data via sql query and some of the data has commas. Output File must be csv and I cannot update the data in the db (as it is used by other application). Example table FavoriteThings Person VARCHAR2(25), Favorite VARCHAR2(100) Sample Data Greta rain drop on... (12 Replies)
Discussion started by: patk625
12 Replies

2. Shell Programming and Scripting

Need code for updating second record to first record in shell scripting

Hi,, I have requirement that i need to get DISTINCT values from a table and if there are two records i need to update it to one record and then need to submit INSERT statements by using the updated value as a parameter. Here is the example follows.. SELECT DISTINCT ID FROM OFFER_GROUP WHERE... (1 Reply)
Discussion started by: Samah
1 Replies

3. Shell Programming and Scripting

Generating & executing the SELECT and UPDATE clause dynamically using shell script

Hi All, I need to write one shell script. The requirement is as follows. a) I am having list the employee names in a file stored in /tmp directory location as below /tmp/emp.txt b) and the contents are as below cat emp.txt ravi raj ram arun c) I need to connect to sybase... (1 Reply)
Discussion started by: Gowtham_giri
1 Replies

4. Shell Programming and Scripting

Shell Script for generating NTIDs(Usernames) from the email ids which exists in MSAD Directory

Hi Everyone, I just need a shell script which automatically gives the list of NT IDs mean the Usernames from the list of email ids. We have around 140 users from AMERICAS,ASIAPACIFIC and EMEA User Directories and we have their email ids.For ex. i have email id called naveen-kumar.dasu@hp.com... (7 Replies)
Discussion started by: naveen.dasu
7 Replies

5. Shell Programming and Scripting

Print record count of a file using shell script

HI, I need to print the record count of a file using shell script. If the record count of a file excluding header and trailer record if greater than zero then print 'Record count of a file is xxxx records'. If the record count is zero print 'zero records' Thanks Mahendra (1 Reply)
Discussion started by: mmeda
1 Replies

6. Shell Programming and Scripting

Millions of record using shell script

I wanna create a millions of record using shell scripts.. the scenario is somehow like this,,,i want to create the record for these fields in my database..id,name,addrs,sales. So can you please give a overall idea for how to achieve it? (9 Replies)
Discussion started by: shankarpanda003
9 Replies

7. Shell Programming and Scripting

Extract record in shell script,so far using awk without success

hello, I require help in the following. There is a report similar to the ones shown here. I need to do couple of things 10/07 12:47 0003210042 R TENN ANVISER0 DELF96A.V1.O.TENRREG 120710 124657 U 00000 DELFOR 1 4331 10/07 12:47 0003210043 S TENN... (2 Replies)
Discussion started by: rakeshv
2 Replies

8. Shell Programming and Scripting

Need script to remove millions of tmp files in /html/cache/ directory

Hello, I just saw that on my vps (centOS) my oscommerce with a seo script has created millions of tmp files inside the /html/cache/ directory. I would need to remove all those files (millions), I tried via shell but the vps loads goes to very high and it hangs, is there some way to do a... (7 Replies)
Discussion started by: andymc1
7 Replies

9. UNIX for Dummies Questions & Answers

A shell script or software for generating random passwords

Hi, Is there an shell script/batch file to genarate random passwords which expires after a stipulated time period? Please suggest a software which does this for AIX and windows both else. Thanks. (5 Replies)
Discussion started by: dwiravi
5 Replies

10. Shell Programming and Scripting

Parsing record into multiple records in Shell Script

Hi, I am trying to parse a very long record in a text file into multiple records by checking ADD, DELETE, or MODIFY field value in a shell script. Input # File name xyz.txt ADD|N000|8015662|DELETE|N001|9915662|MODIFY|N999|85678 Output ADD|N000|8015662| DELETE|N001|9915662|... (8 Replies)
Discussion started by: naveed
8 Replies
Login or Register to Ask a Question