Shell script is taking more than 3 hrs to execute


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script is taking more than 3 hrs to execute
# 1  
Old 11-16-2007
Shell script is taking more than 3 hrs to execute

Hi



I am doing a process of converting all the values of key column into a row, for eg



Key col1 col2

1 1 1

1 2 1

1 1 3

1 3 1

2 1 1

2 1 2



What I am doing in the Script is convert this data into



1(key)|1:2:1:3 (All Col1 values),1:1:3:1(all col2 values)

2(key)|1:1,1:2

TO achieve this i am using two while loops and 4 If else loops

Now in my production the number of columns are 4 and the number records in the input file are 0.2 million (2 lac) and this script is taking more than 3 hrs to run.

Any idea on how to minimize the execution time?
# 2  
Old 11-16-2007
No error checking ! Not complete !

Using hash, this should be super-fast ! Smilie

Code:
#! /opt/third-party/bin/perl

open(FILE, "<", "r");

while(<FILE>) {
  next if(/^$/);
  chomp;
  my @arr = split(/ /);
  my @val = split(/,/, $fileHash{$arr[0]});
  $val[0] .= (":" . $arr[1]);
  $val[1] .= (":" . $arr[2]);
  $val[0] .= ("," . $val[1]);
  $val[0] =~ s/,:/,/;
  $val[0] =~ s/^://;
  $fileHash{$arr[0]} = $val[0];
}

close(FILE);

foreach my $k ( keys %fileHash ) {
  print "$k $fileHash{$k}\n";
}

exit 0

# 3  
Old 11-18-2007
awk

Hi,

This one should be ok.

input:
Code:
1 1 1

1 2 1

1 1 3

1 3 1

2 1 1

2 1 2

3 1 1

4 2 1

4 1 3

1 3 1

2 1 1

2 1 2

optput:
Code:
2|1:1:1:1|1:1:1:1
3|1|1
4|2:1|2:1
1|1:2:1:3:3|1:2:1:3:3

code:
Code:
awk '
{
if (NF>1)
{
	col[$1]=$1
	if (col1[$1]=="")
		col1[$1]=$2
	else
		col1[$1]=sprintf("%s:%s",col1[$1],$2)
	if (col2[$1]=="")
		col2[$1]=$2
	else
		col2[$1]=sprintf("%s:%s",col2[$1],$2)
}
}
END{

for (i in col)
	print i"|"col1[i]"|"col2[i]
}' filename

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problem in taking input Shell script

I was writing a shell script, where i need to run a command through script and then take input from user and later terminate it. i am not sure how to take input in a proper format immediately after a command. example: below command line is used to import some data in to database (LDAP)... (3 Replies)
Discussion started by: Mridul17
3 Replies

2. Shell Programming and Scripting

Batch script to execute shell script in UNIX server

Hi team, My requirement is to transfer pdf files from windows machine to unix server and then from that unix server we should sftp to another server. I have completed the first part i.e From windows to using to unix server with the help of psftp.exe code: psftp user@host -pw password <... (1 Reply)
Discussion started by: bhupeshchavan
1 Replies

3. Shell Programming and Scripting

Taking information from a postgres sql query and putting it into a shell script array

I have a postgres sql statement that is the following: select age from students; which gives me the entries: Age --- 10 15 13 12 9 14 10 which is about 7 rows of data. Now what I would like to do with this is use a shell script to create an array age. As a results... (3 Replies)
Discussion started by: JSNY
3 Replies

4. Shell Programming and Scripting

Adding a cron job that will execute everyday at 14:50 hrs

Hi I want to execute a cron job everyday at 14:50 hrs. I have a script like this: TMP_FILE="/tmp/tmpcron.txt" RES=0 /usr/bin/crontab -l >> $TMP_FILE ADD_JOB="50 14 * * * /opt/mypath/scriptname.sh" grep "scriptname.sh" $TMP_FILE >> /dev/null JOB_NOT_EXIST=$? if test $JOB_NOT_EXIST... (2 Replies)
Discussion started by: saurabhkoar
2 Replies

5. Shell Programming and Scripting

Been working since 25+ hrs: Bash Script to rename files supposedly direct but difficult to execute

:wall::wall::wall: Hi I have horrible script below, need help in renaming ls -l output into new filename format: Desired output: cp -pv original_path/.* newDirectory/owner_of_file.%dd%mm%y.file_extension.first_8_characters_of_original_filename localuser@localuser:~ vi... (3 Replies)
Discussion started by: wolf@=NK
3 Replies

6. Shell Programming and Scripting

Dos batch script to execute unix shell script

Can anyone help me with a dos batch script to execute a shell script residing in an unix server. I am not able to use ssh. Thanks in advance (2 Replies)
Discussion started by: Shri123
2 Replies

7. Shell Programming and Scripting

Shell Script to execute Oracle query taking input from a file to form query

Hi, I need to query Oracle database for 100 users. I have these 100 users in a file. I need a shell script which would read this User file (one user at a time) & query database. For instance: USER CITY --------- ---------- A CITY_A B CITY_B C ... (2 Replies)
Discussion started by: DevendraG
2 Replies

8. UNIX for Advanced & Expert Users

command taking lot of time to execute

Hi, I am running the following command, and it tries to delete some dn from ldap, however, it takes lot of time before it finally request LDAP server to delete it. I am trying to find why it is taking lot of time. Could you anyone help me in this regard. I have copies the pstack output, and... (3 Replies)
Discussion started by: john_prince
3 Replies

9. Shell Programming and Scripting

How to use ssh execute other shell script on other host (shell script include nohup)?

i want use ssh on the host01 to execute autoexec.sh on the host02 like following : host01> ssh host02 autoexec.sh autoexec.sh include nohup command like follwing : nohup /home/jack/deletedata.sh & after i execute ssh host02 autoexec.sh one the host01. i can't found deletedata.sh... (1 Reply)
Discussion started by: orablue
1 Replies

10. Shell Programming and Scripting

Execute via crontab taking username/password from file

Dear All Here is the details what i want to achieve from shell scripts I have a sever where 5 databases are created. which i having diffrent SID's. Now i want to execute some SQL queries on each one of the databases. (SQL Query is same).That i want to acheive via crontab Now each one of the... (2 Replies)
Discussion started by: jhon
2 Replies
Login or Register to Ask a Question