Data manipulation with Awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Data manipulation with Awk
# 1  
Old 08-24-2009
Data manipulation with Awk

Hello guys,

I'm a new member here and I need some help with the Awk application. I'm using it through the Terminal app of OSX (I'm a Mac user).

I have a huge file with a large amount of data (rows of 3D cartesian coordinates). The data is typically like the following example (actually, the data is made of several thousands of "curves" defined by a list of points in 3D) :

Quote:
2.344 1.453 -1.345
1.234 3.134 3.123
2.566 1.345 2.344
-1.234 -2.333 -4.555
1.344 1.453 -1.345 # End of curve
3.234 4.134 6.123
2.566 4.345 3.344
-1.234 -2.333 -4.555
-7.344 1.453 -5.345 # End of curve
1.234 -3.134 3.123
6.566 5.345 8.344
-1.234 2.333 4.555 # End of curve
I need Awk to read this input file, and write out the number of lines for each "curve" (with some arbitrary symbol in front of it), followed by a list of line numbers. From the data above, Awk should output something like this :

Quote:
NumberOfLines 5
0 1 2 3 4

NumberOfLines 4
5 6 7 8

NumberOfLines 3
9 10 11
Take note that the first line is numbered as "0".
How can I do that ? Someone has an idea ? SmilieSmilie

The only data manipulation I know to do with Awk is like this example :

Quote:
cat path_to_input | awk '{ print $1 "\t" $2 "\t" $3 "\t"}' > path_to_output
Thanks, and sorry for my bad English.
# 2  
Old 08-24-2009
Try this:

Code:
awk '
{s=n?s " " c++:c++; n++}
/# End/{print "NumberOfLines " n; print s "\n"; n=0; s =""}
' file

# 3  
Old 08-24-2009
Hi!!

Franklin 52, could you explain, your command line....i don't understand the part...

Code:
{s=n?s " " c++:c++;

Thanks by advance.
# 4  
Old 08-24-2009
Quote:
Originally Posted by protocomm
Hi!!

Franklin 52, could you explain, your command line....i don't understand the part...

Code:
{s=n?s " " c++:c++;

Thanks by advance.
I've used a conditional operator, the form is :

Code:
expr ? action1 : action2

If expr is true, return action1 else return action2

Code:
s=n?s " " c++:c++

Explanation:

if n != 0 then s = s " " c++ else s = c++

Regards

Last edited by Franklin52; 08-24-2009 at 04:21 PM.. Reason: Correction of explanation: if n != 0 instead of if n = 0
# 5  
Old 08-24-2009
Great, thanks...
# 6  
Old 08-24-2009
Quote:
Originally Posted by Franklin52
Try this:

Code:
awk '
{s=n?s " " c++:c++; n++}
/# End/{print "NumberOfLines " n; print s "\n"; n=0; s =""}
' file

OMG !! It worked like a charm !

THANK YOU SO MUCH !!!
# 7  
Old 08-24-2009
Code:
awk '/End of/{
if(flag==0){
  flag=1
  print "NumberOfLines"NR
  for(i=1;i<=NR;i++){
    printf i-1" "
  }
  pre=NR
}
else{
  num=NR-pre
  print "NumberOfLines"num
  for(i=pre+1;i<=NR;i++){
    printf i-1" "
  }
  pre=NR
}
print ""
}'

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Data manipulation, Please help..

Hello, I have a huge set of data that needs to be reformatted. Here is a simple example to explain the process. I have number n=5 and a input with many numbers separated with comma: ... (11 Replies)
Discussion started by: liuzhencc
11 Replies

2. UNIX for Dummies Questions & Answers

Data Manipulation

Dear Sir, I have file input RGR001|108.28|-2.86489|100-120|RANGGAR RGR002|108.071|-2.69028|80-100|RANNGAR RGR003|108.168|-2.97053|50-80|RANNGAR RGR007|108.192722222|-2.766138889|0-50|RANGGARI want to create files by joining each rows with each rows below Output as below ... (4 Replies)
Discussion started by: radius
4 Replies

3. UNIX for Dummies Questions & Answers

Data/date manipulation

Hallo Team, I need your help. I would like to change field9 format to yyyy-mm-dd it should be for example 2013-11-16 instead of 20131116 0780112843,0873599381,E,ISOL,ZAR,0.0035,O,1,20131116,4373200,0.21 0733001720,0873516499,E,ISOL,ZAR,0.0035,O,1,20131116,4331600,0.21... (3 Replies)
Discussion started by: kekanap
3 Replies

4. UNIX for Dummies Questions & Answers

Data manipulation

Hallo Team, I need to manipulate existing data file. Have a look at current data and expected data: Current Data: 27873517141 27873540000 27873515109 27873517140 27873540001 27873540000 27873501343 27873540000 27873517140 27873511292 27873645989 27873540000 27873540000... (7 Replies)
Discussion started by: kekanap
7 Replies

5. Shell Programming and Scripting

Data manipulation using shell

Dear all I have a dataset (in text format,delimited by tab) which have 100 variables (say, var0-var99) and more than 100,000 observations. I want to do the following: 1. for variable var0-var49, I want to add "00" in front of each data (for example, "1" would become "001") 2. for variable... (8 Replies)
Discussion started by: littlewenwen
8 Replies

6. Shell Programming and Scripting

Help with data manipulation script

Y,T,,H05,6,6,0,0 -> TH05_6 D,5,BT,B -> BT_KIOSK P,KQC222 -> KQC222 G,B,2 -> BRANI_GATE_2 fileA TPM658 Y,T,,H05,6,6,0,0 TPM110 D,5,BT,B TPM136 P,KQC222 TPM180 P,BQC913 TPM575 Y,B,,T05,14,14,0,0 IPM760 G,B,2 TPM011 I need to use second column $1,$2,$3,$4..... if first char... (6 Replies)
Discussion started by: ment0smintz
6 Replies

7. Shell Programming and Scripting

awk data subsets manipulation

Hi, I'm working on a data file with the following structure val1,val2,flag 214.7332983,979.0259,1 12.87435571,205.7679,1 1.365976384,19.01616,1 44.08584096,205.7679,2 7.034721792,383.8778,2 189.5685503,979.0259,2 1.96352032,19.01616,2 where the field 'flag' identifies different... (10 Replies)
Discussion started by: Linoleum
10 Replies

8. Shell Programming and Scripting

Data manipulation from one file

HI all i have a file consisting of following numbers 0000 0000 0000 0000 0000 1010 0000 0100 0000 0000 0000 1111 0000 1010 0000 0100 (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

9. UNIX for Dummies Questions & Answers

Data Manipulation

Hello I am currently having problems in mapulating a certain file which contains vaious data. Belos is a sample content Event=<3190> Client IP=<151.111.11.143> DNS=<abc.sbc.com> TransCount=<139> Client IP=<150.222.133.163> DNS=<xyz.yuu.com> TransCount=<3734> Event=<3120> Client... (11 Replies)
Discussion started by: khestoi
11 Replies

10. UNIX for Dummies Questions & Answers

data manipulation script

I have a folder called {homedata} Within this folder there are 12 subfolders 200601.......200612 Within each subfolder there are 8 sets of files Each filename commences with A B C D E F G or H, so {filename}* can be used. I am trying to write a script which will from the top level go... (1 Reply)
Discussion started by: grinder182533
1 Replies
Login or Register to Ask a Question