Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

FASTEST way to loop a script 10k times

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting FASTEST way to loop a script 10k times
# 1  
Old 03-05-2012
FASTEST way to loop a script 10k times
Is there any FASTEST way to loop a script 10k times

my script works likes this

Code:
c-randomnumbers-script -i input1.bed -g g19 -e DB >> output1

I need to run this 10k times by using consecutive outputs to get my final output i.e, output10000

Code:
c-random-script -i input1.bed -g g19 -e DB >> output1
c-random-script -i output1 -g g19 -e DB >> output2
...................
c-random-script -i output9999 -g g19 -e DB >> output10000

# 2  
Old 03-05-2012
not fastest, maybe shortest.

Code:
seq 1 10000 |while read line
do
  c-randomnumbers-script -i output${line} -g g19 -e DB >> output${line}
done

# 3  
Old 03-06-2012
From your text I get the impression you are concerned only with the final file and not the 9,999 intermediate ones, and from that it doesn't seem that you are appending to existing output files as your example suggests, but are creating new intermediate files to be used only for the next input. If I assume correctly then this should work:

Code:
input=input1.bed
for (( i = 0; i < 10000; i++ ))
do
    c-random-script -i $input -g g19 -e DB >output
    mv output last_output
    input=last_output
done
mv last_output final-desired-name

The speed will depend completely on the time it takes "random script" to execute 10K times.
Last edited by agama; 03-06-2012 at 12:12 AM.. Reason: buggered code tag of all things!!
# 4  
Old 03-06-2012
More clear explanation and a small change in the request.


Code:
c-randomnumbers-script -i input1.bed -g g19 -e DB >> outputX1
intesect-script -a outputX1 -b input2.bed -wa -wa  -f1 |awk '!a[$4]++' |awk '{print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$10}' >> outputY1
rm outputX1

# I need to run this 10k times by using same input to get final 10k outputs i.e outputY1 to outputY10000

# Then output a table having 4th column and 10th column in a file named stats.txt and calculate mean and standard deviation of 7th column 0f 10k outputY* files

Code:
c-randomnumbers-script -i input1.bed -g g19 -e DB >> outputX1;
intesect-script -a outputX1 -b input2.bed -wa -wa  -f1 |awk '!a[$4]++' |awk '{print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$10}' >> outputY1;
rm outputX1 ;

c-randomnumbers-script -i input1.bed -g g19 -e DB >> outputX2;
intesect-script -a outputX2 -b input2.bed -wa -wa  -f1 |awk '!a[$4]++' |awk '{print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$10}' >> outputY2;
rm outputX2;

.........

c-randomnumbers-script -i input1.bed -g g19 -e DB >> outputX10000
intesect-script -a outputX10000 -b input2.bed -wa -wa  -f1 |awk '!a[$4]++' |awk '{print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$10}' >> outputY10000
rm outputX10000

stats.txt
Code:
4thcolumn    10thcol_total    mean    SD
id1_outputY1-id_outputY10k    2000    20    2

# 5  
Old 03-06-2012
Code:
seq 1 10000 |while read line
do
  c-randomnumbers-script -i input1.bed -g g19 -e DB >> outputX${line}
  intesect-script -a outputX${line} -b input2.bed -wa -wa  -f1 |awk '!a[$4]++ {print $1, $2, $3, $4, $5, $6, $10}' OFS="\t" >> outputY${line};
  rm outputX${line}
done

You need explain the second rules more detail, I don't see 10th column in your sample stats.txt
Code:
Then output a table having 4th column and 10th column in a file named stats.txt and calculate mean and standard deviation of 7th column 0f 10k outputY* files

# 6  
Old 03-06-2012
Yeah sorry for explaining clear. I spend some time explain thing more clearly and also included some coded that i needed to add.

I need to calculate the mean and standard deviation of the 7th column keys by using output1 to output10000.
For ex:

Code:
output1
col1 col2 col3 col4 col5 col6 col7
abc1 10 20 x.a.1 0 + key1
abc1 15 20 x.a.2 0 + key1
abc1 10 20 y.a.1 0 + key2
abc1 10 20 c.a.1 0 + key2
abc1 10 20 r.a.1 0 + key2
abc1 50 80 m.a.1 0 - key_3

output2
col1 col2 col3 col4 col5 col6 col7
abc1    5    20    x.a.1    0    +    key1
abc1    105    200    x.a.2    0    +    key1
abc1    150    250 y.a.1    0    +    key2
abc1    130    240    c.a.1    0    +    key3
abc1    130    220    r.a.1    0    +    key3
abc1    500    800    m.a.1    0    -    key_3

output10k
....................................

The count the keys based on output

Code:
output1_key1    2
outut1_key2    3
output1_key_3    1
output1_key1    2
outut1_key2    3
output1_key_3    1
output10k_key..........

Then calculate mean using
Code:
awk ‘{s+=$2} END{print “Sum: “s, “\nNumber of lines: “NR, “\nAverage: “s/(NR)}’

Then calculate standard deviation using
Code:
awk ‘{sum+=$2;sumsq+=$2*$2} END {print sqrt(sumsq/NR – (sum/NR)^2)}’

final output
Code:
key    mean    s.d
key1
key2
key_3
......
key_16

---------- Post updated at 06:37 AM ---------- Previous update was at 12:44 AM ----------

I wrote more clear explanation. Please let me know it is clear enough. Thanx
Last edited by quincyjones; 03-06-2012 at 07:36 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Loop to execute 2 times and send an email alert

After the successful start of server, it should check the status again, if it is not running ,it should go through the loop for 2 times. Even after two times of execution if still the server is not running it should send an alert email. Please help (1 Reply)
Discussion started by: thomas9192
1 Replies

2. Shell Programming and Scripting

Reset while loop to loop same file multiple times

Hi, I want to read file multiple times. Right now i am using while loop but that is not working. ex. While read line do while read line2 do echo stmt1 #processing some data based on data., done < file2.txt done < file1.txt # This will have 10... (4 Replies)
Discussion started by: tmalik79
4 Replies

3. UNIX for Advanced & Expert Users

System call failed with 127 .. after 500 times when called in loop

Hi Experts, I have a code like this. ===== #include.... int main() { int count = 0; while(1){ printf("\n Interation number is: %d \n ",count); rv = system(" test.sh > log.txt " ); if (-1 == rv) { printf("Could not generate static log: error... (12 Replies)
Discussion started by: binnyjeshan
12 Replies

4. Programming

Problem with implementing the times() function in C (struct tms times return zero/negative values)

Hello, i'm trying to implement the times() function and i'm programming in C. I'm using the "struct tms" structure which consists of the fields: The tms_utime structure member is the CPU time charged for the execution of user instructions of the calling process. The tms_stime structure... (1 Reply)
Discussion started by: g_p
1 Replies

5. Shell Programming and Scripting

The loop was executed $count times

#!/bin/sh count=0 for i in 2 4 6 do echo "i is $i" count='expr $count + 1' done echo "The loop was executed $count times" with these scripts my output is : i is 2 i is 4 i is 6 The loop was executed expr $count + 1 times What should I do to get the value instead of 'expr... (17 Replies)
Discussion started by: ymwong
17 Replies

6. Shell Programming and Scripting

Breaking the files as 10k recs. per file

Hi, I have a code as given below Set -A _Category="A\ B\ C" for _cat in ${_Category} do sed -e "s:<TABLE_NAME>:${_cat}:g" \ -e "s:<date>:${_dt}:g" \ ${_home}/skl/sq1.sql >> ${_dest}/del_${_dt}.sql fi ... (4 Replies)
Discussion started by: mr_manii
4 Replies

7. Shell Programming and Scripting

script for comparing the times.

Hi, My below output file looks like this. Backup status at Tue Jul 7 03:30:07 EDT 2009 CMLP Job Name Last Start Last End ST Run Pri/Xit ____________________________ ____________________ ____________________ __ _______ ___ tnx415uu 07/06/2009 22:01:41 ----- RU 5547800/1 tnp415uu 07/06/2009... (2 Replies)
Discussion started by: intiraju
2 Replies

8. UNIX for Dummies Questions & Answers

how to cut selected 10k lines continuosly

I have a source file having 20lacs lines i want to cut first 1 lac lines from source file and redirect to other file (i.e after redirecting the output to other file the source file should have only 19 lacs count ). Pls help me in getting solution. Thanks in advance. (9 Replies)
Discussion started by: vamshi
9 Replies

9. Shell Programming and Scripting

Loop to check for file up to 3 times

Please forgive this I think rather basic question. I have been away from UNIX for a very long time and am in need of some help. I need to be able to check for the existance of a specific file name say 'file.dat' in a particular location If the file exists then run a second process (at... (2 Replies)
Discussion started by: JohnCrump
2 Replies

10. Shell Programming and Scripting

Help script for login times

I am new to shellscript . PLease help me how can I write the following script. $ who ray pts/0 aug 31 01:18 ( 65.169.28.200 ) ray pts/1 sep 2 02:28 ( 65.169.28.200 ) bob pts/3 sep 2 02:31 ( 65.169.28.201 ) when run the command who |./ script , the script should... (3 Replies)
Discussion started by: LAY
3 Replies
Login or Register to Ask a Question