Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk processing of variable number of fields data file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk processing of variable number of fields data file
# 1  
Old 11-15-2013
awk processing of variable number of fields data file
Hy!

I need to post-process some data files which have variable (and periodic) number of fields. For example, I need to square (data -> data*data) the folowing data file:
Code:
 -5.34281E-28 -3.69822E-29  8.19128E-29  9.55444E-29  8.16494E-29  6.23125E-29
  4.42106E-29  2.94592E-29  1.84841E-29  1.09271E-29  6.08599E-30  3.19287E-30
  1.57732E-30  7.33449E-31  3.20866E-31  1.31982E-31  5.10059E-32  1.85021E-32
  6.29190E-33  2.00262E-33  5.95280E-34  1.64748E-34
 -5.34281E-28 -3.69822E-29  8.19128E-29  9.55444E-29  8.16494E-29  6.23125E-29
  4.42106E-29  2.94592E-29  1.84841E-29  1.09271E-29  6.08599E-30  3.19287E-30
  1.57732E-30  7.33449E-31  3.20866E-31  1.31982E-31  5.10059E-32  1.85021E-32
  6.29190E-33  2.00262E-33  5.95280E-34  1.64748E-34
 -5.34281E-28 -3.69822E-29  8.19128E-29  9.55444E-29  8.16494E-29  6.23125E-29
  4.42106E-29  2.94592E-29  1.84841E-29  1.09271E-29  6.08599E-30  3.19287E-30
  1.57732E-30  7.33449E-31  3.20866E-31  1.31982E-31  5.10059E-32  1.85021E-32
  6.29190E-33  2.00262E-33  5.95280E-34  1.64748E-34

The data is fitted into 6 columns, but depending on the requested precission, one might found he/she has also some lines with less number of columns.

For the processing, some newbie knowledge of awk suffice. In my case I use something like
Code:
awk '{printf("%12.8G %12.8G %12.8G %12.8G %12.8G %12.8G\n", $1*$1, $2*$2, $3*$3, $4*$4, $5*$5, $6*$6)}' initial.data > final.data

which produces something like this
Code:
2.8545619E-55 1.3676831E-57 6.7097068E-57 9.1287324E-57 6.6666245E-57 3.8828477E-57
1.9545772E-57 8.6784446E-58 3.4166195E-58 1.1940151E-58 3.7039274E-59 1.0194419E-59
2.4879384E-60 5.3794744E-61 1.0295499E-61 1.7419248E-62 2.6016018E-63 3.423277E-64
3.9588006E-65 4.0104869E-66 3.5435828E-67 2.7141904E-68            0            0

Question #1: how can one eliminate the "0"s which awk produces? I've tried sed
Code:
sed 's/        0    /             /g' <final.data >almost.final.data

but I can't remove the last 0 from each smaller line (i.e. fewer columns with real data); in this case I obtain something like this:
Code:
2.8545619E-55 1.3676831E-57 6.7097068E-57 9.1287324E-57 6.6666245E-57 3.8828477E-57
1.9545772E-57 8.6784446E-58 3.4166195E-58 1.1940151E-58 3.7039274E-59 1.0194419E-59
2.4879384E-60 5.3794744E-61 1.0295499E-61 1.7419248E-62 2.6016018E-63 3.423277E-64
3.9588006E-65 4.0104869E-66 3.5435828E-67 2.7141904E-68                         0

Question #2: How can I stop awk process a non existing data from a column? (in my case, the 5th and 6th "fields" from every 4 columns-only lines)

I thank you for your help!
# 2  
Old 11-15-2013
You could check explicitly check for every 4th line:
Code:
awk '
NR%4==0 { 4 columns stuff }
NR%4!=0 { 6 columns stuff}'

(EDIT: Or check NF==4/NF==6, as was suggested in a briefly-lived post Smilie)


Or just loop around the number of fields you actually have in each line:
Code:
awk '
{
   for (i=1;i<=NF;i++) {
      printf ("%12.8G ", $i*$i)
   }
   printf "\n"
}'

Last edited by CarloM; 11-15-2013 at 01:00 PM..
# 3  
Old 11-15-2013
Try also
Code:
awk '{print  $1*$1, $2*$2, $3*$3, $4*$4, $5?$5*$5:"", $6?$6*$6:""}' OFMT="%14.8G" file

# 4  
Old 11-15-2013
Yet another possibility:
Code:
 awk '{for(i=1; i<=NF; i++) $i*=$i}1' CONVFMT="%14.8G" file

# 5  
Old 11-15-2013
Thank you guys!

I've only implemented Scrutinizers suggestion, and it works well for me.

For CarloM: (I'm not an awk expert, but ...) can you please tell me where should I state in your second code what file should be processed? Thank you!
# 6  
Old 11-16-2013
You can redirect stdin as you did with with your sed command, or just specify the filename as in the other suggestions.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Setting the number of fields using awk

Hi Guys, I've obviously had a senior moment here, what I'm trying to do is set the number of fields to 35 in a csv these should be appended to the end of the line. But what I'm getting is:- Source Data $ head out_file_01.txt N1000,024,2809003,,,3,DYNAMIC AVLEASE INC,PO BOX... (10 Replies)
Discussion started by: gull04
10 Replies

2. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

3. Shell Programming and Scripting

[ksh88 and awk] Number of fields with a value.

Hi, With: # VALUES="one~two~~~" # echo $VALUES | awk 'BEGIN {FS="~"} {print NF}' 5 I can determine the number of fields. How to determine the number of fields with a value ? In this case 2. Thanks in advance, ejdv (6 Replies)
Discussion started by: ejdv
6 Replies

4. Shell Programming and Scripting

Data processing using awk

Hello, I have some bitrate data in a csv which is in an odd format and is difficult to process in Excel when I have thousands of rows. Therefore, I was thinking of doing this in bash and using awk as the primary application except that due to its complication, I'm a little stuck. ... (24 Replies)
Discussion started by: shadyuk
24 Replies

5. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6. Below is an... (5 Replies)
Discussion started by: ambroze
5 Replies

6. Shell Programming and Scripting

number of fields in a text file as a variable - perl

I am looking for perl code to get following o/p. If a line has more than 7 fields then value in field 7 onwards is BHA_GRP1, BHA_GRP2, BHA_GRP3, BHA_GRP4 etc. Here is example of what I am trying to achieve. INPUT File: VAH NIC_TYPE CONFIG SIZE_GB PILO KOM BHA_GRP1 BHA_GRP2 BHA_GRP3...... 2... (1 Reply)
Discussion started by: dynamax
1 Replies

7. Shell Programming and Scripting

Help with data processing, maybe awk

I have a file, first 5 columns are very normal, like "1107",106027,71400,"Y","BIOLOGY",, however, the 6th columns, the user can put comments, anything, just any characters, like new line, double quote, single quote, whatever from the keyboard, like"Please load my previous SOM597G course content in... (3 Replies)
Discussion started by: freelong
3 Replies

8. Shell Programming and Scripting

awk sed cut? to rearrange random number of fields into 3 fields

I'm working on formatting some attendance data to meet a vendors requirements to upload to their system. With some help on the forums here, I have the data close. But they've since changed what they want. The vendor wants me to submit three fields to them. Field 1 is the studentid field,... (4 Replies)
Discussion started by: axo959
4 Replies

9. Shell Programming and Scripting

Parse apart strings of comma separated data with varying number of fields

I have a situation where I am reading a text file line-by-line. Those lines of data contain comma separated fields of data. However, each line can vary in the number of fields it can contain. What I need to do is parse apart each line and write each field of data found (left to right) into a file.... (7 Replies)
Discussion started by: 2reperry
7 Replies

10. Shell Programming and Scripting

AWK - printing certain fields when field order changes in data file

I'm hoping someone can help me on this. I have a data file that greatly simplified might look like this: sec;src;dst;proto 421;10.10.10.1;10.10.10.2;tcp 426;10.10.10.3;10.10.10.4;udp 442;10.10.10.5;10.10.10.6;tcp sec;src;fac;dst;proto 521;10.10.10.1;ab;10.10.10.2;tcp... (3 Replies)
Discussion started by: eric4
3 Replies
Login or Register to Ask a Question