Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Modify log files to get uniq data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Modify log files to get uniq data
# 1  
Old 08-24-2009
Modify log files to get uniq data
Hello,

I have a log file that has following output as below.

Code:
LAP.sun5 CC
LAP.sun5 CQ
perl.sun5 CC
perl.sun5 CQ
TSLogger.sun5 CC
TSLogger.sun5 CQ
TSLogger.sun5 KR
WAS.sun5 CC
WAS.sun5 MT
WAS.sun5 CQ

I want to output to be in the way below, i tried using awk but could not do it.
Code:
LAP.sun5 CC CQ
perl.sun5 CC CQ
TSLogger.sun5 CC CQ KR
WAS.sun5 CC MT CQ

Please help . how do i do that

Thanks
Adsi
Last edited by asirohi; 08-24-2009 at 05:12 PM..
# 2  
Old 08-24-2009
Code:
#!/bin/sh
for entry in `awk '{ print $1 }' < your.log.file | sort -u`
do
  string=""
  for result in `egrep "^$entry " your.log.file | awk '{ print $2 }'`
  do
    string="$result $string"
  done
  echo "$entry $string"
done

Exact grep syntax will depend on your version of grep available, egrep for Solaris, grep -e for most others, grep for the rest
# 3  
Old 08-24-2009
Quote:
Originally Posted by asirohi
...
I want to output to be in the way below, i tried using awk but could not do it.
Code:
LAP.sun5 CC CQ
perl.sun5 CC CQ
TSLogger.sun5 CC CQ KR
WAS.sun5 CC MT CQ

...
Code:
$ 
$ cat data.txt
LAP.sun5 CC
LAP.sun5 CQ
perl.sun5 CC
perl.sun5 CQ
TSLogger.sun5 CC
TSLogger.sun5 CQ
TSLogger.sun5 KR
WAS.sun5 CC
WAS.sun5 MT
WAS.sun5 CQ
$ 
$ awk '{if (prev == ""){printf("%s",$0)}
        else if($1 == prev){printf(" %s",$2)}
             else {printf("\n%s",$0)} prev=$1
       } END {printf "\n"}' data.txt
LAP.sun5 CC CQ
perl.sun5 CC CQ
TSLogger.sun5 CC CQ KR
WAS.sun5 CC MT CQ
$ 
$

tyler_durden
# 4  
Old 08-25-2009
Hello Tyler,

Thanks for the wonderful explaination, that was superb. I have a question if(prev == "") and i dont want to display any thing what do i do. I asked this question because in the log files there are $1 fields that occur only once and i dont want to display them. For example
HTML Code:
[COLOR=Red]Linix CQ[/COLOR]
LAP.sun5 CC
LAP.sun5 CQ
perl.sun5 CC
perl.sun5 CQ
TSLogger.sun5 CC
TSLogger.sun5 CQ
TSLogger.sun5 KR
WAS.sun5 CC
WAS.sun5 MT
WAS.sun5 CQ
[COLOR=Red]Sun.log CC[/COLOR]

So the output should be as below and should leave $1 field 
if it has no previous value.

LAP.sun5 CC CQ
perl.sun5 CC CQ
TSLogger.sun5 CC CQ KR
WAS.sun5 CC MT CQ


Code:
$ awk '{if (prev == ""){printf("%s",$0)}
        else if($1 == prev){printf(" %s",$2)}
             else {printf("\n%s",$0)} prev=$1
       } END {printf "\n"}' data.txt

Last edited by asirohi; 08-25-2009 at 10:48 AM.. Reason: code tags, PLEASE!
# 5  
Old 08-25-2009
A different approach
Code:
awk 'NF{_[$1]=$0}END{for(i in _)print _[i]}' file | sort

# 6  
Old 08-25-2009
Hello Danmer,

That does not work, did it work for you?
# 7  
Old 08-25-2009
Works for me, try to use GNU awk (gawk), New awk (nawk) or POSIX awk (/usr/xpg4/bin/awk).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need help in awk: running a loop with one column and segregate data 4 each uniq value in that field

Hi All, I have a file like this(having 2 column). Column 1: like a,b,c.... Column 2: having numbers. I want to segregate those numbers based on column 1. Example: file. a 5 b 9 b 620 a 710 b 230 a 330 b 1910 (4 Replies)
Discussion started by: Raza Ali
4 Replies

2. Shell Programming and Scripting

Combine data from two files base on uniq data

File 1 ID Name Po1 Po2 DD134 DD134_4A_1 NN-1 L_0_1 DD134 DD134_4B_1 NN-2 L_1_1 DD134 DD134_4C_1 NN-3 L_2_1 DD142 DD142_4A_1 NN-1 L_0_1 DD142 DD142_4B_1 NN-2 L_1_1 DD142 DD142_4C_1 NN-3 L_2_1 DD142 DD142_3A_1 NN-41 L_3_1 DD142 DD142_3A_1 NN-42 L_3_2 File 2 ( Combination of... (1 Reply)
Discussion started by: pareshkp
1 Replies

3. UNIX for Dummies Questions & Answers

Modify Column Data using Shell Script

HI Guys, Input :- P081 wr1 12p0d5: 22.8 P081 wr1 12p2d18: 23.1 P149 wr1 1pxcud6/port_0_dev_7: 20.4 P149 wr1 1pxcud4/port_1_dev_10: 22.4 OutputP081 wr1 120 22.8 P081 wr1 122 23.1 P149 wr1 10 20.4 P149 wr1 11 22.4 In in First two line delete p and after d untill : In Last two line... (4 Replies)
Discussion started by: pareshkp
4 Replies

4. Shell Programming and Scripting

Filtering data using uniq and sed

Hello, Does anyone know an easy way to filter this type of file? I want to get everything that has score (column 2) 100.00 and get rid of duplicates (for example gi|332198263|gb|EGK18963.1| below), so I guess uniq can be used for this? gi|3379182634|gb|EGK18561.1| 100.00... (6 Replies)
Discussion started by: narachaid
6 Replies

5. Shell Programming and Scripting

How can I modify my script to include or substitute missing data?

Let me start off by saying I am a self taught sometimes scripter so what you will see below won't be pretty. I have created a script to parse through a file with a large amount of data and simply pull out what I need. In doing this I create several files and then paste them together in order to... (2 Replies)
Discussion started by: fsanchez
2 Replies

6. UNIX for Dummies Questions & Answers

Finding and Extracting uniq data in multiple files

Hi, I have several files that look like this: File1.txt Data1 Data2 Data20 File2.txt Data1 Data5 Data10 File3.txt Data1 Data2 Data17 File4.txt (6 Replies)
Discussion started by: Fahmida
6 Replies

7. Shell Programming and Scripting

Dynamically modify data in file?

Hiya fellas, I have a data file which contains these fields... EMPID EMPNAME GENDER DESIGNATION AGE SALARY this datafile looks like this.. Now, i'm making a script which will allow me to edit anything on execution. ie. First it will ask which EMPID is to be modified. and then it... (3 Replies)
Discussion started by: rushyang
3 Replies

8. Shell Programming and Scripting

Using uniq on log files

I have this log file which I need to count the number of repeated line and do some manipulation. test.log: June 3 03:33:38 test 1 June 3 10:31:22 test 2 June 3 10:32:22 test 2 June 3 10:33:22 test 3 June 3 10:33:22 test 3 June 3 10:34:22 test 4 June 3 10:35:22 test 5 ... (4 Replies)
Discussion started by: jazzaddict
4 Replies

9. Shell Programming and Scripting

Help needed with Sort and uniq data

Hi All, After Sorting directories and files i have got following output as below, now i only want the strings common in them, so the actual output should be as below in the bottom. How do i do that? Thanks -adsi File to be modified:- Common Components for ----> AA... (4 Replies)
Discussion started by: asirohi
4 Replies

10. Shell Programming and Scripting

How to replicate data using Uniq or awk

Hi, I have this scenario; where there are two classes:- apple and orange. 1,2,3,4,5,6,apple 1,1,0,4,2,3,apple 1,3,3,3,3,4,apple 1,1,1,1,1,1,orange 1,2,3,1,1,1,orange Basically for apple, i have 3 entries in the file, and for orange, I have 2 entries. Im trying to edit the file and find... (5 Replies)
Discussion started by: ahjiefreak
5 Replies
Login or Register to Ask a Question