AWK - printing certain fields when field order changes in data file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AWK - printing certain fields when field order changes in data file
# 1  
Old 04-15-2008
Question AWK - printing certain fields when field order changes in data file

I'm hoping someone can help me on this. I have a data file that greatly simplified might look like this:

Code:
sec;src;dst;proto
421;10.10.10.1;10.10.10.2;tcp
426;10.10.10.3;10.10.10.4;udp
442;10.10.10.5;10.10.10.6;tcp
sec;src;fac;dst;proto
521;10.10.10.1;ab;10.10.10.2;tcp
525;10.10.10.5;ac;10.10.10.6;tcp
522;10.10.10.3;ab;10.10.10.4;udp
535;10.10.10.5;ac;10.10.10.6;tcp
...

Periodically throughout the file is a header line and the lines underneath that are the actual data where the fields correspond to the header line. However, sometimes the order of the fields change, and different fields are used, but that change is always marked by a new header line. The new header lines can be anywhere in the data file. Fields might be next to each other at one point in the file, then separated in a later part of the file.

I'd like to be able to produce a simplified output of just a few of the fields in the data file. For instance, I'd like to extract the src, dst, and proto from the example above. src is normally field 2, but dst is field 3 and then changes to field 4. My desired output would look something like this:

Code:
src;dst;proto
10.10.10.1;10.10.10.2;tcp
10.10.10.3;10.10.10.4;udp
10.10.10.5;10.10.10.6;tcp
10.10.10.1;10.10.10.2;tcp
10.10.10.5;10.10.10.6;tcp
10.10.10.3;10.10.10.4;udp
10.10.10.5;10.10.10.6;tcp

I've worked with AWK quite a bit and know how to work with field numbers, if/then, etc, but I can't figure out how to change a field number to a new value as directed by the header line.

Can anyone help me? I'd sure appreciate any advice. Is AWK the right tool to do this with?
# 2  
Old 04-15-2008
Hi eric, I think AWK is the right stuff for you. Try to evaluate some fields (or the entire line) to look for the headers so you know that next lines follow that pattern until a new header comes. At least headers are static, aren't them?

Good luck!
# 3  
Old 04-15-2008
I think I figured it out:

Code:
awk '
BEGIN {FS=";"} 
/;src;/{
for (num=1;num<=NF;num++) {if ($num == "src") fieldsrc=num}; 
for (num=1;num<=NF;num++) {if ($num == "dst") fielddst=num}; 
for (num=1;num<=NF;num++) {if ($num == "proto") fieldproto=num};
} 

!/;src;/{print $fieldsrc";"$fielddst";"$fieldproto}
'

seems to work for this example:

10.10.10.1;10.10.10.2;tcp
10.10.10.3;10.10.10.4;udp
10.10.10.5;10.10.10.6;tcp
10.10.10.1;10.10.10.2;tcp
10.10.10.5;10.10.10.6;tcp
10.10.10.3;10.10.10.4;udp
10.10.10.5;10.10.10.6;tcp

Now if I can figure it out for the real world data ...
# 4  
Old 04-15-2008
# default fields: 'src;dst;proto'
nawk -f eric.txt myDataFile.txt

# fields order : 'proto;sec;src'
nawk -v fields='proto;sec;src' -f eric.txt myDataFile.txt

eric.awk:
Code:
BEGIN {
  FS=OFS=";"

  if (fields=="") fields="src;dst;proto"

  n=split(fields, fieldsA, FS)

  PATheader="[;]*src[;]*"
}

FNR==1 { print fields }
$0 ~ PATheader {
   for(i=1; i<=NF; i++)
      header[$i]=i
   next
}

{
   for(i=1; i<=n; i++)
     printf("%s%c", $header[fieldsA[i]], (i==n) ? ORS : OFS)
}

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk to clean up input file, printing both fields

In the f1 file below I am trying to clean it up removing lines the have _tn_ in them. Next, removing the characters in $2 before the ninth /. Then I remove the ID_(digit- always 4). Finally, the charcters after and including the first _. It is curently doing most of it but the cut is removing $1... (5 Replies)
Discussion started by: cmccabe
5 Replies

2. UNIX for Beginners Questions & Answers

Continued trouble matching fields in different files and selective field printing ([g]awk)

I apologize in advance, but I continue to have trouble searching for matches between two files and then printing portions of each to output in awk and would very much appreciate some help. I have data as follows: File1 PS012,002 PRQ 0 1 1 17 1 0 -1 3 2 1 2 -1 ... (7 Replies)
Discussion started by: jvoot
7 Replies

3. Shell Programming and Scripting

awk Selective printing of fields

Hi Gurus, I have following input file. I tried multiple awk combinations to print selected columns without success. HEX ID Name ver FLRGT Start Time Total Shared End Date ----- -------- --- ------ ------------------------ -------------- -------... (4 Replies)
Discussion started by: shunya
4 Replies

4. Shell Programming and Scripting

awk processing of variable number of fields data file

Hy! I need to post-process some data files which have variable (and periodic) number of fields. For example, I need to square (data -> data*data) the folowing data file: -5.34281E-28 -3.69822E-29 8.19128E-29 9.55444E-29 8.16494E-29 6.23125E-29 4.42106E-29 2.94592E-29 1.84841E-29 ... (5 Replies)
Discussion started by: radudownload
5 Replies

5. Shell Programming and Scripting

How to print 1st field and last 2 fields together and the rest of the fields after it using awk?

Hi experts, I need to print the first field first then last two fields should come next and then i need to print rest of the fields. Input : a1,abc,jsd,fhf,fkk,b1,b2 a2,acb,dfg,ghj,b3,c4 a3,djf,wdjg,fkg,dff,ggk,d4,d5 Expected output: a1,b1,b2,abc,jsd,fhf,fkk... (6 Replies)
Discussion started by: 100bees
6 Replies

6. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6. Below is an... (5 Replies)
Discussion started by: ambroze
5 Replies

7. Shell Programming and Scripting

Putting values into order in a field using awk

Hi, I am using UBUNTU 12.04. I have a dataset as follows: Column#1 Column#2 Column#3 .... Column#50 1 154878 1 145145 2 189565 2 454121 ... (5 Replies)
Discussion started by: Homa
5 Replies

8. UNIX for Dummies Questions & Answers

printing fields in reverse order

command/script(apart from awk) to print the fields in reverse order that is last field has to come first and so on and first field has to go last Input store-id date sale ............. ............. ... (3 Replies)
Discussion started by: tsurendra
3 Replies

9. Shell Programming and Scripting

need help with awk in printing the fields in a file

hi all i get a file from a server and i do not know how many fields that file will contain. i need to cut from the second column of the file to the last, irrespective of how many fields are there. is there any way to make the awk command dynamic for fetching from the second to the last... (4 Replies)
Discussion started by: sais
4 Replies

10. Shell Programming and Scripting

printing select fields in awk

Hi, I want to print certain fields from my data file depending on certain conditions. Somebody pls let me know how to send it to awk. The command below is the one which I want to use in a shell script and this prints fine cat ./datafile.dat | grep -i $SEARCH_STR | awk -F: '{ print $1 $2 $3... (5 Replies)
Discussion started by: maverix
5 Replies
Login or Register to Ask a Question