Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Improve script and get new output file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Improve script and get new output file
# 8  
Old 06-14-2015
Quote:
Originally Posted by jiam912
Dear RudiC
Thanks For your comments, the script works Fine But slow.
I will attach examples or input and output Files, they are big so i cant post Here,
Thanks again.
Don't post huge files; post small, representative sample input files and the corresponding, desired output files.

It looks like your script is creating lots of files that aren't really needed (and some of them are removed at the end of the script). Which file(s) do you really want?
This User Gave Thanks to Don Cragun For This Post:
jiam912
# 9  
Old 06-14-2015
Dear Don Cragun and RudiC.

Here are the sample of files i have

file1
Code:
X     0       01   54589.00  20593.001 1793 18681  54764.00  20626.00  20776.001
X     0       01   54589.00  20601.001 1877 19561  54764.00  20626.00  20784.001
X     0       01   54589.00  20609.001 1961 20441  54764.00  20626.00  20792.001
X     0       01   54589.00  20617.001 2045 21321  54764.00  20626.00  20800.001
X     0       01   54589.00  20625.001 2129 22201  54764.00  20626.00  20808.001
X     0       01   54589.00  20633.001 2213 23081  54764.00  20626.00  20816.001
X     0       01   54589.00  20641.001 2297 23961  54764.00  20626.00  20824.001
X     0       01   54589.00  20649.001 2381 24841  54764.00  20626.00  20832.001
X     0       01   54589.00  20657.001 2465 25721  54764.00  20626.00  20840.001
X     0       01   54589.00  20665.001 2549 26601  54764.00  20626.00  20848.001
X     0       01   54589.00  20673.001 2633 27481  54764.00  20626.00  20856.001
X     0       01   54589.00  20681.001 2717 28361  54764.00  20626.00  20864.001
X     0       01   54589.00  20689.001 2801 29241  54764.00  20626.00  20872.001
X     0       01   54589.00  20697.001 2885 30121  54764.00  20626.00  20880.001
X     0       01   54589.00  20705.001 2969 31001  54764.00  20626.00  20888.001
X     0       01   54589.00  20713.001 3053 31881  54764.00  20626.00  20896.001
X     0       01   54589.00  20721.001 3137 32761  54764.00  20626.00  20904.001
X     0       01   54589.00  20729.001 3221 33641  54764.00  20626.00  20912.001
X     0       01   54589.00  20737.001 3305 34521  54764.00  20626.00  20920.001
X     0       01   54589.00  20745.001 3389 35401  54764.00  20626.00  20928.001
X     0       01   54589.00  20753.001 3473 36281  54764.00  20626.00  20936.001

file2
Code:
S5458920749 483857.9   2558238.5                           8.9  5458920745 2611 000 Ex
S5458920717 483161.4   2557837.0                           8.8  5458920721 2611 000 Ex
S5458920639 481469.5   2556860.3                           6.9  5458920641 2611 000 Ex
S5458920611 480873.0   2556515.5                           1.7  5458920593 2611 000 Ex
S5458920615 480952.0   2556561.0                           1.7  5458920609 2611 000 Ex
S5458920613 480914.2   2556539.4                           1.7  5458920601 2611 000 Ex

desired output
Code:
X     0       01   54589.00  20611.003 1793 18681  54764.00  20626.00  20776.001
X     0       01   54589.00  20613.003 1877 19561  54764.00  20626.00  20784.001
X     0       01   54589.00  20615.003 1961 20441  54764.00  20626.00  20792.001
X     0       01   54589.00  20617.001 2045 21321  54764.00  20626.00  20800.001
X     0       01   54589.00  20625.001 2129 22201  54764.00  20626.00  20808.001
X     0       01   54589.00  20633.001 2213 23081  54764.00  20626.00  20816.001
X     0       01   54589.00  20639.003 2297 23961  54764.00  20626.00  20824.001
X     0       01   54589.00  20649.001 2381 24841  54764.00  20626.00  20832.001
X     0       01   54589.00  20657.001 2465 25721  54764.00  20626.00  20840.001
X     0       01   54589.00  20665.001 2549 26601  54764.00  20626.00  20848.001
X     0       01   54589.00  20673.001 2633 27481  54764.00  20626.00  20856.001
X     0       01   54589.00  20681.001 2717 28361  54764.00  20626.00  20864.001
X     0       01   54589.00  20689.001 2801 29241  54764.00  20626.00  20872.001
X     0       01   54589.00  20697.001 2885 30121  54764.00  20626.00  20880.001
X     0       01   54589.00  20705.001 2969 31001  54764.00  20626.00  20888.001
X     0       01   54589.00  20713.001 3053 31881  54764.00  20626.00  20896.001
X     0       01   54589.00  20717.003 3137 32761  54764.00  20626.00  20904.001
X     0       01   54589.00  20729.001 3221 33641  54764.00  20626.00  20912.001
X     0       01   54589.00  20737.001 3305 34521  54764.00  20626.00  20920.001
X     0       01   54589.00  20749.003 3389 35401  54764.00  20626.00  20928.001
X     0       01   54589.00  20753.001 3473 36281  54764.00  20626.00  20936.001

Using file2.txt, I need to replace the string matching in file 1,
example
file2.txt
Code:
S5458920611 480873.0   2556515.5                           1.7  5458920593 2611 000 Ex

colum 65 to 74 search for this value in file1.txt colum 20 to 24 and 30 to 34 = 5458920593, and replace with value in column 2 to 11 (file2.txt)
file1.txt
Code:
X     0       01   54589.00  20593.001 1793 18681  54764.00  20626.00  20776.001

Output file
Code:
X     0       01   54589.00  20611.003 1793 18681  54764.00  20626.00  20776.001

Also when the change is done need to replace column 38 with value 3 instead of 1.

The changes need to produce a new file keeping the original file as it is.

Appreciate your help.
# 10  
Old 06-14-2015
And how does this fit to the problem in post#1?
# 11  
Old 06-14-2015
Dear RudiC.

This is the reason why I did this part of the script
## create Dbase ---
Code:
awk '{\
	ori_line=substr($5,1,5);\
	ori_point=substr($5,6,5);\
	off_line=substr($1,2,5);\
	off_point=substr($1,7,5);\
	printf (file1.txt %5d.00   %5d.00 %5d.00   %5d.00\n",ori_line,ori_point,off_line,off_point)}' file2.txt > $file

to get the same format of file1.txt and to match the values from file2.txt
# 12  
Old 06-14-2015
With not inconsiderable guesswork I tried to reconstruct the underlying file structure for four files:
Code:
spread_1.x01
spread_2.x01
spread_3.x01
spread_4.x01
1offb1-Sx.sps
2offb1-Sx.sps
3offb1-Sx.sps
4offb1-Sx.sps

Is that close to reality? If yes, this bash script might do what you need

Code:
#!/bin/bash
read -p "from: " fsw
read -p "to  : " lsw

awk '
FNR==1          {delete CHGARR 
                 swath=FILENAME
                 sub (/^.*_/,  "", swath)
                 sub (/\..*$/, "", swath)
                 while (1 == getline LINE < (swath"offb1-Sx.sps"))
                        {split (LINE, TMP)
                         CHGARR[TMP[5]]=TMP[1]
                        }
                }

                {IX = (int($4) int($5))}

IX in CHGARR    {sub ($4, sprintf("%8.2f",  substr(CHGARR[IX], 2, 5)))
                 sub ($5, sprintf("%8.2f3", substr(CHGARR[IX], 7, 5)))
                }

                {print $0 > FILENAME".new"}

' $(eval echo spread_{$fsw..$lsw}.x01)

Please be aware that {$fsw..$lsw} is a bashism and does not necessarily exist in other shells.

If fsw and lsw are single digit numbers only, you could simplify the file selection to spread_[$fsw-$lsw].x01


Results: with the change file 1offb1-Sx.sps (file2)
Code:
S5458920611 480873.0   2556515.5                           1.7  5458920593 2611 000 Ex

, the line from data file spread_1.x01 (file1)
Code:
X     0       01   54589.00  20593.001 1793 18681  54764.00  20626.00  20776.001

becomes spread_1.x01.new
Code:
X     0       01   54589.00  20611.003 1793 18681  54764.00  20626.00  20776.001

Give it a shot and report back.
Last edited by RudiC; 06-14-2015 at 10:25 AM..
This User Gave Thanks to RudiC For This Post:
jiam912
# 13  
Old 06-14-2015
Dear RudiC.

You are really fantastic, and professional , the script, works perfectly and very fast .
My script was doing same process in 2 minutes , while with your script the job ends in 15 sec .
I am very grateful for your help. Thanks for your time and support.
I do not understand many things in the script , as { split ( TMP LINE } . But I will try to figure out it slowly .
If you have time, kindly could explain the procedure of the script.
Thank you very much again.
# 14  
Old 06-14-2015
man awk is your friend.

Please be aware that a detailed and precise specification with input and output samples and directory/file structure would have saved quite some time and effort.
These 2 Users Gave Thanks to RudiC For This Post:
Don Cragun jiam912
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Improve script

Gents, Is there the possibility to improve this script to be able to have same output information. I did this script, but I believe there is a very short code to get same output here my script awk -F, '{if($10>0 && $10<=15) print $6}' tmp1 | sort -k1n | awk '{a++} END { for (n in a )... (23 Replies)
Discussion started by: jiam912
23 Replies

2. Shell Programming and Scripting

How to improve an script?

Gents. I have 2 different scripts for the same purpose: raw2csv_1 Script raw2csv_1 finish the process in less that 1 minute raw2csv_2 Script raw2csv_2 finish the process in more that 6 minutes. Can you please check if there is any option to improve the raw2csv_2. To finish the job... (4 Replies)
Discussion started by: jiam912
4 Replies

3. Shell Programming and Scripting

Improve sftp script

Dear all, I have written two scripts to transfer files to another server outside the company. One is a batch script , and the other script calls the batch script, send the files and archive the file sent. The problem is, that I want to get the list of files which have been uploaded the the... (10 Replies)
Discussion started by: arrals_vl
10 Replies

4. UNIX for Dummies Questions & Answers

How to improve the performance of this script?

Hi , i wrote a script to convert dates to the formate i want .it works fine but the conversion is tkaing lot of time . Can some one help me tweek this script #!/bin/bash file=$1 ofile=$2 cp $file $ofile mydates=$(grep -Po '+/+/+' $ofile) # gets 8/1/13 mydates=$(echo "$mydates" | sort |... (5 Replies)
Discussion started by: vikatakavi
5 Replies

5. Shell Programming and Scripting

Looking to improve the output of this awk one-liner

I have the following awk one-liner I came up with last night to gather some data. and it works pretty well (apologies, I'm quite new with awk, and don't know how to format this pretty-printed). You can see the output with it. awk '{if ($8 == 41015 && $21 == "requests") arr+=$20;if ($8 == 41015... (3 Replies)
Discussion started by: DeCoTwc
3 Replies

6. Shell Programming and Scripting

Var Check Script (Help improve if possible)

I am working on a script to check the var on all of my systems. Can someone help me fix it to work better or give me suggestions. #!/bin/ksh IN=/path/to/list_of_workstations.txt while read hostnames do if ping $hostnames 1 | grep alive > /dev/null then percent=`ssh -q... (3 Replies)
Discussion started by: whotippedmycow
3 Replies

7. Shell Programming and Scripting

Want to improve the performance of script

Hi All, I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately. Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search... (6 Replies)
Discussion started by: poweroflinux
6 Replies

8. Shell Programming and Scripting

Improve the performance of a shell script

Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very... (10 Replies)
Discussion started by: apsprabhu
10 Replies

9. Shell Programming and Scripting

Any way to improve performance of this script

I have a data file of 2 gig I need to do all these, but its taking hours, any where i can improve performance, thanks a lot #!/usr/bin/ksh echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')" function showHelp { cat << EOF >&2 syntax extreme.sh FILENAME Specify filename to parse EOF... (3 Replies)
Discussion started by: sirababu
3 Replies

10. Shell Programming and Scripting

Can I improve this script ???

Hi all, Still a newbie and learning as I go ... as you do :) Have created this script to report on disc usage and I've just included the ChkSpace function this morning. It's the first time I've read a file (line-by-bloody-line) and would like to know if I can improve this script ? FYI - I... (11 Replies)
Discussion started by: Cameron
11 Replies
Login or Register to Ask a Question