How to improve an script?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to improve an script?
# 1  
Old 12-15-2014
How to improve an script?

Gents.

I have 2 different scripts for the same purpose:

Code:
raw2csv_1

Script raw2csv_1 finish the process in less that 1 minute

Code:
raw2csv_2

Script raw2csv_2 finish the process in more that 6 minutes.

Can you please check if there is any option to improve the
Code:
raw2csv_2

. To finish the job quickly.

Attached.

Code:
Scripts
Input file
Output file generated.

Thanks for your support.
# 2  
Old 12-15-2014
Most users will not download your attached files and extract.
1) Takes time and effort
2) Potentially damaging viruses

You may want to post the scripts, or sections that concern you, as pasted text instead.
# 3  
Old 12-15-2014
script raw2csv_1

Code:
#!/bin/csh

cd /user/

#!/bin/csh -f
date
set filename = `echo $1 | sed -e 's/\./ /' | awk '{ print $1 }'`
awk ' RS = "# ===== " { if ( NR == 6) {print $0} }' $1 |\
awk ' RS = "]"  { if (NR >0) { print $0} }' | sed -e 's/
//g' |\
sed -e 's/:/ /g' | grep -v Report | sed -e '/^ *$/d' |\
sed -e 's/\t/ /g' | sed -e 's/"//g' | grep -v "====" |\
grep -v "                       [0-9]" |\
grep -v "^    Live_Seis_Channels" |\
 awk '{ for (i=1;i<=1;i++) { printf ("%s ",  $i)}; printf ","}' > $filename"_1".tmp
  sed -e 's/ ===== (.*) =====/#-#-#-#/g' $1 | awk ' RS = "##-#-#-#" { if ( NR > 1 ) {print $0} }' |\
  awk ' RS = "]"  { print $0 }' |\
  sed -e 's/:/ /g' | grep -v Report |\
  grep -v Report |\
  grep -v "^    Live_Seis_Channels" |\
  grep -v "^ *          " |\
  sed -e '/^ *$/d' | sed -e 's/\t/ /g' |\
  sed -e 's/"//g' |\
  awk '{\
         for (i=2;i<=NF;i++) {\
                                                  printf ("%s ", $i)\
                             };\
                             printf ("%s",",");\
                             } END { printf "\n" } ' | sed -e 's/# (ms)//g' | sed -e 's/# (msec)//g' | sed -e 's/#//g' | sed -e 's/
//g' > $filename"_2".tmp
echo ""

cat  $filename"_1".tmp  $filename"_2".tmp | sed -e  's/,===============  ,/\n/g'   > $filename.csv

date

rm -f "$filename"_1.tmp "$filename"_2.tmp

raw2csv_2

Code:
#!/bin/bash

cd /user/
date

filename=${1%.*}
sed 's/# ===== (.*) =====/#-#-#-#/g' $1 | \
awk '
NR>1 {
    gsub(/[\r\"\]]/,"")
    gsub(/[:\t]/," ")
    gsub(" *\n", "\n")
    gsub("\n *", "\n")
    gsub("\n#^[\n]*\n", "\n")
    gsub("\n\n+", "\n")
    printf "%s", $0
}' RS="#-#-#-#" | egrep -v '(====|Report|^[0-9]|^Live_Seis_Channels)' | \
awk -F"\n" -vRS="" 'NR>0{
  for(i=1;i<=NF;i++) {
     H=$i
     gsub(" .*","",H)
     gsub(H" *","",$i)
     gsub("# [(](ms|msec)[)]","",$i)
     gsub("#","",$i)
     V[NR]=V[NR]"  ,"$i
     if(NR==2)HD=HD" ,"H
  }
}
END{ print substr(HD,3)
  for(i=1;i<=NR;i++) print substr(V[i], 4)
}' > "${filename}.csv"
date

# 4  
Old 12-15-2014
Try this; adapting/extending the header line will include the respective fields in your .csv file:
Code:
awk -F: 'BEGIN                  {HD="Version,Exploitation_Mode,Filter_Type,Aux_Nb_Trace,Seis_Nb_Trace,Total_Nb_Trace,Nb_Of_Dead_Seis_Channels,Nb_Of_Live_Seis_Channels,Dead_Seis_Channels"
                                 print HD
                                 HDCnt=split(HD,HDArr,",")
                                 NXTREC="Observer_Report" 
                                 HDCM=","HD","
                                }

                                {gsub (/[\t ]*|\*/, "", $1); sub (/^[\t ]*/, "", $2); sub (/[\t ]*$/,"", $2)}

         $1 == NXTREC && PR     {for (i=1; i<=HDCnt; i++) printf "%s,", RES[HDArr[i]]
                                 printf "%d\n", NR
                                 delete RES
                                }
         $1 == NXTREC           {PR=1}
         HDCM ~ "," $1 ","      {RES[$1]=$2}


         END                    {for (i=1; i<=HDCnt; i++) printf "%s,", RES[HDArr[i]]
                                 printf "\n"
                                }
        ' /tmp/342.raw

There's no error checking included nor the conversions like "msec" -> "" etc. that you have in your posted scripts.

---------- Post updated at 18:40 ---------- Previous update was at 15:48 ----------

I had to correct the script for data fileds that contain ":" like the Date entry:
Code:
awk -F: 'BEGIN                  {HD="Version,Exploitation_Mode,Filter_Type,Date,Aux_Nb_Trace,Seis_Nb_Trace,Total_Nb_Trace,Nb_Of_Dead_Seis_Chan
                                 print HD
                                 HDCnt=split(HD,HDArr,",")
                                 NXTREC="Observer_Report"
                                 HDCM=","HD","
                                }

                                {gsub (/[\t ]*|\*/, "", $1)}

         $1 == NXTREC && PR     {for (i=1; i<=HDCnt; i++) printf "%s,", RES[HDArr[i]]
                                 printf "%d\n", NR
                                 delete RES
                                }
         $1 == NXTREC           {PR=1}
         HDCM ~ "," $1 ","      {T=$1; sub ($1 "[^:]*:[\t ]*", "", $0); sub (/[\t ]*$/, "", $0); RES[T]=$0}


         END                    {for (i=1; i<=HDCnt; i++) printf "%s,", RES[HDArr[i]]
                                 printf "\n"
                                }
        ' OFS=":" /tmp/342

And, make sure you remove the DOS <CR> line separators!
This User Gave Thanks to RudiC For This Post:
# 5  
Old 12-15-2014
Hello RudiC,

Thanks I will try. it
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Improve script

Gents, Is there the possibility to improve this script to be able to have same output information. I did this script, but I believe there is a very short code to get same output here my script awk -F, '{if($10>0 && $10<=15) print $6}' tmp1 | sort -k1n | awk '{a++} END { for (n in a )... (23 Replies)
Discussion started by: jiam912
23 Replies

2. Shell Programming and Scripting

Improve script and get new output file

Gents, Using the following script, I got the changes as desired in the output file called (spread_2611.x01.new). Complete file as input (spread_2611.x01). Can you please have a look to my script and improve it please. :b: Also I would like to I get a additional selecting only the records... (21 Replies)
Discussion started by: jiam912
21 Replies

3. Shell Programming and Scripting

Improve sftp script

Dear all, I have written two scripts to transfer files to another server outside the company. One is a batch script , and the other script calls the batch script, send the files and archive the file sent. The problem is, that I want to get the list of files which have been uploaded the the... (10 Replies)
Discussion started by: arrals_vl
10 Replies

4. UNIX for Dummies Questions & Answers

How to improve the performance of this script?

Hi , i wrote a script to convert dates to the formate i want .it works fine but the conversion is tkaing lot of time . Can some one help me tweek this script #!/bin/bash file=$1 ofile=$2 cp $file $ofile mydates=$(grep -Po '+/+/+' $ofile) # gets 8/1/13 mydates=$(echo "$mydates" | sort |... (5 Replies)
Discussion started by: vikatakavi
5 Replies

5. Shell Programming and Scripting

Var Check Script (Help improve if possible)

I am working on a script to check the var on all of my systems. Can someone help me fix it to work better or give me suggestions. #!/bin/ksh IN=/path/to/list_of_workstations.txt while read hostnames do if ping $hostnames 1 | grep alive > /dev/null then percent=`ssh -q... (3 Replies)
Discussion started by: whotippedmycow
3 Replies

6. UNIX for Dummies Questions & Answers

[please] improve my shell/SQL*Plus script

Hi We generate with PL/SQL *.csv files, archive them and mail to the customer. Here is my script (Solaris 10, ksh): #!/bin/ksh # Unix Shell Script Structure for PL/SQL queries with SQL*Plus . ~/.profile scriptdir=/opt/ora/scripts queryname1=example... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

7. Shell Programming and Scripting

Want to improve the performance of script

Hi All, I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately. Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search... (6 Replies)
Discussion started by: poweroflinux
6 Replies

8. Shell Programming and Scripting

Improve the performance of a shell script

Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very... (10 Replies)
Discussion started by: apsprabhu
10 Replies

9. Shell Programming and Scripting

Any way to improve performance of this script

I have a data file of 2 gig I need to do all these, but its taking hours, any where i can improve performance, thanks a lot #!/usr/bin/ksh echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')" function showHelp { cat << EOF >&2 syntax extreme.sh FILENAME Specify filename to parse EOF... (3 Replies)
Discussion started by: sirababu
3 Replies

10. Shell Programming and Scripting

Can I improve this script ???

Hi all, Still a newbie and learning as I go ... as you do :) Have created this script to report on disc usage and I've just included the ChkSpace function this morning. It's the first time I've read a file (line-by-bloody-line) and would like to know if I can improve this script ? FYI - I... (11 Replies)
Discussion started by: Cameron
11 Replies
Login or Register to Ask a Question

Featured Tech Videos