Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

IF awk in a while read line-loop

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting IF awk in a while read line-loop
# 1  
Old 05-16-2012
IF awk in a while read line-loop
Hi

As a newbe in scripting, i struggle hard with my first script.
What i want to do is, bringing data of two files together.



file1:
Code:
....
05/14/12-04:00:00 41253 4259 5135 5604 5812 5372 
05/14/12-04:10:00 53408 5501 6592 7402 7354 6639 
05/14/12-04:20:00 58748 6037 7292 8223 8357 7492 
05/14/12-04:30:00 57192 7824 7987 7235 
05/14/12-04:40:00 56901 5585 6721 7684 7839 7166 
05/14/12-05:00:00 45642 4361 5049 5593 5684 5467 
....

lag --> .. missing the data from 04:50
file1 has sometimes breaks [datalags] and the lag should be pastet free.

file2:
Code:
....
05/14/12-04:00 4.04 191.22
05/14/12-04:10 4.36 200.91
05/14/12-04:20 3.55 206.60
05/14/12-04:30 2.85 213.91
05/14/12-04:40 2.23 213.21
05/14/12-04:50 1.82 224.51
05/14/12-05:00 2.04 262.31
....


my idea was ...
comparing the timestamp TS1={date=$1 and time=$2 of file1} with the timestamp of file2 TS2.
And if TS1 == TS2, then the awk-specified values of file1 should be pastet line by line at file2 > file3

The result should be
Code:
....
05/14/12-04:00 4.04 191.22 41253 4259 5135 5604 5812 5372 
05/14/12-04:10 4.36 200.91 53408 5501 6592 7402 7354 6639 
05/14/12-04:20 3.55 206.60 58748 6037 7292 8223 8357 7492 
05/14/12-04:30 2.85 213.91 57192 7824 7987 7235 
05/14/12-04:40 2.23 213.21 56901 5585 6721 7684 7839 7166 
05/14/12-04:50 1.82 224.51 
05/14/12-05:00 2.04 262.31 45642 4361 5049 5593 5684 5467 
 
....

Code:
 
 
while read line
do
UDaps=`echo $line | awk -F'-' '{print $1}'`
UZaps=`echo $line | awk -v 'FS=-\"?|\"? \"?|\"?$' '{print $2}'`
TS2="$UDaps " " $UZaps" 
 
...??
done < $Home/METE/file2.txt

i try to solve it with "while read line" and check $TS2.
Next step; i want to check the $TS1 --> then compare it with $TS2.
IF it is equal, paste the awk specified values line by line.
Can you please help me?

Thanks in advance!
Ralf
Last edited by radoulov; 05-16-2012 at 11:21 AM.. Reason: Additional code tags.
# 2  
Old 05-16-2012
This is too hard to read, please remove the font tags and use code tags for data and code samples.
Last edited by Franklin52; 05-16-2012 at 08:06 AM..
# 3  
Old 05-16-2012
Code:
# awk -F'[: ]' 'NR==FNR{cc=NR;s=" ";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b," ");if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file1 file2
05/14/12-04:00 4.04 191.22 41253 4259 5135 5604 5812 5372
05/14/12-04:10 4.36 200.91 53408 5501 6592 7402 7354 6639
05/14/12-04:20 3.55 206.60 58748 6037 7292 8223 8357 7492
05/14/12-04:30 2.85 213.91 57192 7824 7987 7235
05/14/12-04:40 2.23 213.21 56901 5585 6721 7684 7839 7166
05/14/12-04:50 1.82 224.51
05/14/12-05:00 2.04 262.31 45642 4361 5049 5593 5684 5467

regards
ygemici
This User Gave Thanks to ygemici For This Post:
IMPe
# 4  
Old 05-16-2012
Hi ygemici!

Thank you for this fast answer.
I'm just a beginner and so it ist hard for me to follow your code, especially in using the arrays.
The sript is stil not working properly and i think the reasons are:


Quote:
Originally Posted by ygemici
Code:
# awk -F'[: ]' 'NR==FNR{cc=NR;s=" ";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b," ");if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file1 file2
05/14/12-04:00 4.04 191.22 41253 4259 5135 5604 5812 5372
05/14/12-04:10 4.36 200.91 53408 5501 6592 7402 7354 6639
05/14/12-04:20 3.55 206.60 58748 6037 7292 8223 8357 7492
05/14/12-04:30 2.85 213.91 57192 7824 7987 7235
05/14/12-04:40 2.23 213.21 56901 5585 6721 7684 7839 7166
05/14/12-04:50 1.82 224.51
05/14/12-05:00 2.04 262.31 45642 4361 5049 5593 5684 5467

i
1. The fildseparator in file1 and file2 is a TAB, so i change the code in ..
Code:
 
awk -F'[: ]' 'NR==FNR{cc=NR;s="\t ";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b,"\t");if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file1 file2

Code:
 
output is ...
....
05/16/12-11:40  8.45    304.44  0.00
05/16/12-11:50  8.74    293.83  0.00
05/16/12-12:00  10.21   297.48  0.00
05/16/12-12:10  10.06   300.32  0.00
05/16/12-12:20  11.25   292.83  0.00
05/16/12-12:30  10.31   294.80  0.00
05/16/12-12:40  10.69   290.08  0.00
05/16/12-12:50  12.38   300.04  0.00
05/16/12-13:00  10.94   283.92  0.00
05/16/12-13:10  11.67   277.96  0.00
05/16/12-13:20  11.52   278.33  0.00
05/16/12-13:30  10.42   281.29  0.00
05/16/12-13:40  12.33   296.57  0.00
05/16/12-13:50  11.54   304.93  0.00
05/16/12-14:00  9.29    305.52  0.00
05/16/12-14:10  9.15    300.11  0.00

....

2. The other point is, that the timeformat in file1 is hh:mm:ss and in file2 the format is hh:mm. i think that is the reason why only the data of file2 comes to printout. I have no idea how to change the script. can you please help me one more time?

Thanks in advance!
Ralf
Last edited by IMPe; 05-16-2012 at 11:28 AM..
# 5  
Old 05-16-2012
Quote:
Originally Posted by IMPe
Hi ygemici!

Thank you for this fast answer.
I'm just a beginner and so it ist hard for me to follow your code, especially in using the arrays.
The sript is stil not working properly and i think the reasons are:




1. The fildseparator in file1 and file2 is a TAB, so i change the code in ..
Code:
 
awk -F'[: ]' 'NR==FNR{cc=NR;s="\t ";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b,"\t");if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file1 file2

Code:
 
output is ...
....
05/16/12-11:40  8.45    304.44  0.00
05/16/12-11:50  8.74    293.83  0.00
05/16/12-12:00  10.21   297.48  0.00
05/16/12-12:10  10.06   300.32  0.00
05/16/12-12:20  11.25   292.83  0.00
05/16/12-12:30  10.31   294.80  0.00
05/16/12-12:40  10.69   290.08  0.00
05/16/12-12:50  12.38   300.04  0.00
05/16/12-13:00  10.94   283.92  0.00
05/16/12-13:10  11.67   277.96  0.00
05/16/12-13:20  11.52   278.33  0.00
05/16/12-13:30  10.42   281.29  0.00
05/16/12-13:40  12.33   296.57  0.00
05/16/12-13:50  11.54   304.93  0.00
05/16/12-14:00  9.29    305.52  0.00
05/16/12-14:10  9.15    300.11  0.00

....

2. The other point is, that the timeformat in file1 is hh:mm:ss and in file2 the format is hh:mm. i think that is the reason why only the data of file2 comes to printout. I have no idea how to change the script. can you please help me one more time?

Thanks in advance!
Ralf
my mistake,try this with little change that the file order Smilie
Code:
# awk -F'[:\t]' 'NR==FNR{cc=NR;s="\t";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b,s);if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file2 file1

This User Gave Thanks to ygemici For This Post:
IMPe
# 6  
Old 05-16-2012
Hi ygemici!

First of all, let me say Thank you for helping me.
I've done what you said, bud it's still not working.
On my Notebook (Linux 3.1.10-1.9-desktop #1 SMP PREEMPT Thu Apr 5 18:48:38 UTC 2012 4x86_64 GNU/Linux || GNU bash, Version 4.2.10(1)-release (x86_64-suse-linux-gnu) ) it is running with the following result

Code:
05/13/12-05:20  01      77831   14.63   0.00
05/13/12-05:30  01      81259   16.59   0.00
05/13/12-05:40  01      88809   20.84   0.00
05/14/12-07:00  00      90181   205.32  0.00
05/14/12-07:09:58       77985   7690    8905    9497    9209    7976    6185    4430    3349    2483    2205    1835    1655    1364   1196     1053    973     813     779     730     632     595     645     631     461     357     259     224     198     188     123    100      76      64      41      32      12      15

the data i want to exclude is pastet into the middle.

On my Desktop-PC (Linux debianPC 2.6.32-5-686 #1 SMP Sun May 6 04:01:19 UTC 2012 i686 GNU/Linux || GNU bash, Version 4.1.5(1)-release (i486-pc-linux-gnu) ) i only get the message...
Code:
awk: line 5: illegal reference to array b

Starting these thread i gave a reduced example of the real fileformat, so that i can - if i understand the code - extend ist by myself.
real-format of file2
Code:
mm/dd/Y-hh:mm \t $2   \t   $3   \t   $4

real-format of file1
Code:
mm/dd/Y-hh:mm:ss \t $2   \t   $3   \t   $4 ... up to ...  \t   $70

I'm still trying to understand the code
Code:
# awk -F'[:\t]' 'NR==FNR{cc=NR;s="\t";f=":";x++;
why is the marked fieldsep. colon and TAB?
i also dont understand the NR==record?

a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
i=4 --> beginning at the 4. column? what ist the s?
NR!=FNR{c=0;for(i=1;i<=x;i++){
Is FNR the record the complete line in the array?
split(a[i],b,s);if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file2 file1

Thanks in advanced!
Ralf
Last edited by IMPe; 05-16-2012 at 06:29 PM..
# 7  
Old 05-18-2012
im still trying to bring two files together. After i clean up my files the pasting line by line according to the date and time field is not working. The Fieldsep is TAB

here again what i want with the original data.
file1
Code:
05/15/12-12:40  7.82    305.30  0.09
05/15/12-12:50  8.38    295.85  0.01
05/15/12-13:00  8.04    298.16  0.01
05/15/12-13:10  9.05    301.95  0.00
05/15/12-13:20  8.09    303.94  0.00
05/15/12-13:30  7.87    314.60  1.13

file2
Code:
05/15/12-12:40  38685   3501    4346    5054    5017    4542    3805    3191    2908    2547    2413    2223    2027    1733    1448    1186    965     788     652     574     507     463     521     468     288     189     115     91      68      39      30      30      19      10      8       4       3       2       0       0       0       0       0       0       0       0       0       1       0       0       0       0       219     978.5   4.99    4       75      56.5    24.6
05/15/12-12:50  44343   3909    5024    5693    5837    4995    3905    3058    2486    2244    2094    1858    1635    1334    1143    908     747     676     533     485     440     395     455     357     267     148     122     132     84      42      28      21      14      11      4       2       5       2       1       1       1       0       1       0       0       1       0       0       0       1       0       0       224     977.3   5.01    4.02    75      55.9    22.2
05/15/12-13:00  41933   3641    4786    5776    5819    5300    4019    2880    2352    2060    1863    1645    1473    1218    1075    862     670     560     524     443     439     397     404     388     254     201     130     133     68      53      37      15      13      15      4       5       5       2       1       1       0       0       3       0       1       2       1       1       1       1       0       0       213     977.3   4.98    3.99    75      55.4    20.7
05/15/12-13:10  49477                                                                   568     494     480     464     471     374     225     185     168     135     80      45      29      20      8       7       8       3       2       1       0       0       0       0       1       0       0       2       0       0       0       0       0       252     977.3   5       3.99    75      55.1    19.7
05/15/12-13:30  67053   6218    8243    9898    9918    8450    6147    4173    3000    2363    2181    1948    1639    1461    1315    1068    917     769     666     589     534     483     505     565     436     257     181     150     125     59      36      23      15      11      3       1       3       0       0       1       1       0       0       0       0       0       0       0       0       0       0       0       324     977.5   4.99    3.99    75      54.8    19.7

and it should be
Code:
05/15/12-12:40  7.82    305.30  0.09 38685   3501    4346    5054    5017    4542    3805    3191    2908    2547    2413    2223    2027    1733    1448    1186    965     788     652     574     507     463     521     468     288     189     115     91      68      39      30      30      19      10      8       4       3       2       0       0       0       0       0       0       0       0       0       1       0       0       0       0       219     978.5   4.99    4       75      56.5    24.6
05/15/12-12:50  8.38    295.85  0.01 44343   3909    5024    5693    5837    4995    3905    3058    2486    2244    2094    1858    1635    1334    1143    908     747     676     533     485     440     395     455     357     267     148     122     132     84      42      28      21      14      11      4       2       5       2       1       1       1       0       1       0       0       1       0       0       0       1       0       0       224     977.3   5.01    4.02    75      55.9    22.2
05/15/12-13:00  8.04    298.16  0.01 41933   3641    4786    5776    5819    5300    4019    2880    2352    2060    1863    1645    1473    1218    1075    862     670     560     524     443     439     397     404     388     254     201     130     133     68      53      37      15      13      15      4       5       5       2       1       1       0       0       3       0       1       2       1       1       1       1       0       0       213     977.3   4.98    3.99    75      55.4    20.7
05/15/12-13:10  9.05    301.95  0.00 49477                                                                   568     494     480     464     471     374     225     185     168     135     80      45      29      20      8       7       8       3       2       1       0       0       0       0       1       0       0       2       0       0       0       0       0       252     977.3   5       3.99    75      55.1    19.7
05/15/12-13:20  8.09    303.94  0.00 
05/15/12-13:30  7.87    314.60  1.13  67053   6218    8243    9898    9918    8450    6147    4173    3000    2363    2181    1948    1639    1461    1315    1068    917     769     666     589     534     483     505     565     436     257     181     150     125     59      36      23      15      11      3       1       3       0       0       1       1       0       0       0       0       0       0       0       0       0       0       0       324     977.5   4.99    3.99    75      54.8    19.7

i tried to solve it with ygemici awk-array-construct but it will not work. --> awk: line 5: illegal reference to array b
Code:
awk -F'[:\t]' 'NR==FNR{cc=NR;s="\t";f=":";x++;
a[x]=$1f$2;;for(i=4;i<=NF;i++)a[x]=a[x]s$i}
NR!=FNR{c=0;for(i=1;i<=x;i++){
split(a[i],b,s);if($1f$2==b[1]){printf "%s",b[1]s$3s$4s;
for(j=2;j<=length(b);j++)printf "%s%c",b[j],s;printf "%s","\n"}
else c++};if(c==cc)print}' file2 file1

i need some help to solve it finally!
Thanks in advance!
IMPe
Last edited by IMPe; 05-18-2012 at 07:46 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

While read line loop

Based on text file: PATH:/media/hdd/Media/Video/title1 FILE:/media/cache/281662-14.jpg PATH:/media/hdd/Media/Video/title2 FILE:/media/cache/281662-15.jpg PATH:/media/hdd/Media/Video/title3 FILE:/media/cache/281662-16.jpg PATH:/media/hdd/Media/Video/title4 FILE:/media/cache/281662-17.jpg... (12 Replies)
Discussion started by: TiedCone
12 Replies

2. Shell Programming and Scripting

Do While Loop + Read From File + assign line to a variable

Hello, I am using below code for reading from a file and assigning the values to a variable , but it is loosing the value after the loop , please suggest to retain the value of the variable after the loop , while IFS=: read -r line do set $dsc=$line echo 'printing line variable ' $line... (1 Reply)
Discussion started by: ParthThakkar
1 Replies

3. Shell Programming and Scripting

Simple while read line loop question

#!/bin/bash count=1 while read line do if (($count > 4)); then awk -v var1="$count" '{printf "%3s%8s%11s%11s%11s\n",var1,$2,$3,$4,$5}' else echo $line fi count=$((count+1)) done < posre_sub.itp > test cat test INPUT: ; position restraints for... (3 Replies)
Discussion started by: origamisven
3 Replies

4. Shell Programming and Scripting

Read file using while loop not reading last line

I have written a script to read the file line by line. It is reading and printing the lines. But it is coming out of loop before reading last line. So I am not able to print last line. How do I solve it. (6 Replies)
Discussion started by: dgmm
6 Replies

5. Shell Programming and Scripting

While loop read line not working

Hi, I am trying to read a file line by line inside of a while loop. This while loop is part of a here document. while read line do ssh -t $2@$remotehost <<REMOTE ls path/to/dir > $path_to_dir while read line1 do echo "LINE --- $line" done... (4 Replies)
Discussion started by: mnanavati
4 Replies

6. Shell Programming and Scripting

While loop read line Issue

Hi I am using while loop, below, to read lines from a very large file, around 400,000 rows. The script works fine until around line 300k but then starts giving incorrect result. I have tried running the script with a smaller data set and it works fine. I made sure to include the line where... (2 Replies)
Discussion started by: saurabhkumar198
2 Replies

7. Shell Programming and Scripting

While read line loop

Hi I'm writing a bash script which will read an input file and look for occurrences of the current user ($USER) executing the script. When i find the occurrence of the username I take that line and append it to a file with a line number and bracket display next to line. The input file has been... (12 Replies)
Discussion started by: BundBash
12 Replies

8. Shell Programming and Scripting

read line in a for loop

Hi All, How can we use read line using the index value of a FOR loop? eg: pt_mstr,pt_status,8 pt_mstr,pt_buyer,8 pt_mstr,pt_sfty_stk,8 pt_mstr,pt_ord_pol,3 pt_mstr,pt_capacity,8 pt_mstr,pt_plan_ord,3 pt_mstr,pt_ord_mult,8 From this file i want to read the line2, 3 and 4 only using a FOR... (3 Replies)
Discussion started by: balajim
3 Replies

9. UNIX for Dummies Questions & Answers

Alternative for a while read line loop

HELLO all :), I have been trying to use a simple while loop to read a file " templist", line by line and perform an action. See the code below. The reason for not using a while read line loop is the for the use of the if condition that wouldn't work. I would appreciate some ideas as this has... (2 Replies)
Discussion started by: kabs
2 Replies

10. Shell Programming and Scripting

Nested while read line loop

Hi, Can anyone please help me: i'm trying to read a file with directory-names , then go to that directory and read another (output) file to perform some tasks per line (second read line in the part of script below). The problem is that after the nested while loop has finished, the first while... (7 Replies)
Discussion started by: Rakker
7 Replies
Login or Register to Ask a Question