Sponsored Content
Full Discussion: Data manipulation with Awk
Top Forums Shell Programming and Scripting Data manipulation with Awk Post 302347334 by Franklin52 on Tuesday 25th of August 2009 11:53:35 AM
Old 08-25-2009
You can remove the double lines first with uniq and pipe the result to the awk command:

Code:
uniq file | awk '
{s=n?s " " c++:c++; n++}
/# End/{print "NumberOfLines " n; print s "\n"; n=0; s =""}
'

Regards
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

data manipulation script

I have a folder called {homedata} Within this folder there are 12 subfolders 200601.......200612 Within each subfolder there are 8 sets of files Each filename commences with A B C D E F G or H, so {filename}* can be used. I am trying to write a script which will from the top level go... (1 Reply)
Discussion started by: grinder182533
1 Replies

2. UNIX for Dummies Questions & Answers

Data Manipulation

Hello I am currently having problems in mapulating a certain file which contains vaious data. Belos is a sample content Event=<3190> Client IP=<151.111.11.143> DNS=<abc.sbc.com> TransCount=<139> Client IP=<150.222.133.163> DNS=<xyz.yuu.com> TransCount=<3734> Event=<3120> Client... (11 Replies)
Discussion started by: khestoi
11 Replies

3. Shell Programming and Scripting

Data manipulation from one file

HI all i have a file consisting of following numbers 0000 0000 0000 0000 0000 1010 0000 0100 0000 0000 0000 1111 0000 1010 0000 0100 (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

4. Shell Programming and Scripting

awk data subsets manipulation

Hi, I'm working on a data file with the following structure val1,val2,flag 214.7332983,979.0259,1 12.87435571,205.7679,1 1.365976384,19.01616,1 44.08584096,205.7679,2 7.034721792,383.8778,2 189.5685503,979.0259,2 1.96352032,19.01616,2 where the field 'flag' identifies different... (10 Replies)
Discussion started by: Linoleum
10 Replies

5. Shell Programming and Scripting

Help with data manipulation script

Y,T,,H05,6,6,0,0 -> TH05_6 D,5,BT,B -> BT_KIOSK P,KQC222 -> KQC222 G,B,2 -> BRANI_GATE_2 fileA TPM658 Y,T,,H05,6,6,0,0 TPM110 D,5,BT,B TPM136 P,KQC222 TPM180 P,BQC913 TPM575 Y,B,,T05,14,14,0,0 IPM760 G,B,2 TPM011 I need to use second column $1,$2,$3,$4..... if first char... (6 Replies)
Discussion started by: ment0smintz
6 Replies

6. Shell Programming and Scripting

Data manipulation using shell

Dear all I have a dataset (in text format,delimited by tab) which have 100 variables (say, var0-var99) and more than 100,000 observations. I want to do the following: 1. for variable var0-var49, I want to add "00" in front of each data (for example, "1" would become "001") 2. for variable... (8 Replies)
Discussion started by: littlewenwen
8 Replies

7. UNIX for Dummies Questions & Answers

Data manipulation

Hallo Team, I need to manipulate existing data file. Have a look at current data and expected data: Current Data: 27873517141 27873540000 27873515109 27873517140 27873540001 27873540000 27873501343 27873540000 27873517140 27873511292 27873645989 27873540000 27873540000... (7 Replies)
Discussion started by: kekanap
7 Replies

8. UNIX for Dummies Questions & Answers

Data/date manipulation

Hallo Team, I need your help. I would like to change field9 format to yyyy-mm-dd it should be for example 2013-11-16 instead of 20131116 0780112843,0873599381,E,ISOL,ZAR,0.0035,O,1,20131116,4373200,0.21 0733001720,0873516499,E,ISOL,ZAR,0.0035,O,1,20131116,4331600,0.21... (3 Replies)
Discussion started by: kekanap
3 Replies

9. UNIX for Dummies Questions & Answers

Data Manipulation

Dear Sir, I have file input RGR001|108.28|-2.86489|100-120|RANGGAR RGR002|108.071|-2.69028|80-100|RANNGAR RGR003|108.168|-2.97053|50-80|RANNGAR RGR007|108.192722222|-2.766138889|0-50|RANGGARI want to create files by joining each rows with each rows below Output as below ... (4 Replies)
Discussion started by: radius
4 Replies

10. Shell Programming and Scripting

Data manipulation, Please help..

Hello, I have a huge set of data that needs to be reformatted. Here is a simple example to explain the process. I have number n=5 and a input with many numbers separated with comma: ... (11 Replies)
Discussion started by: liuzhencc
11 Replies
UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME
uniq -- report or filter out repeated lines in a file SYNOPSIS
uniq [-cdu] [-f fields] [-s chars] [input_file [output_file]] DESCRIPTION
The uniq utility reads the standard input comparing adjacent lines, and writes a copy of each unique input line to the standard output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first. The following options are available: -c Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. -d Don't output lines that are not repeated in the input. -f fields Ignore the first fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adja- cent fields by blanks. Field numbers are one based, i.e. the first field is field one. -s chars Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first fields fields will be ignored. Character numbers are one based, i.e. the first character is character one. -u Don't output lines that are repeated in the input. If additional arguments are specified on the command line, the first such argument is used as the name of an input file, the second is used as the name of an output file. The uniq utility exits 0 on success, and >0 if an error occurs. COMPATIBILITY
The historic +number and -number options have been deprecated but are still supported in this implementation. SEE ALSO
sort(1) STANDARDS
The uniq utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. BSD
January 6, 2007 BSD
All times are GMT -4. The time now is 09:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy