Need help with finding unique string in log file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need help with finding unique string in log file
# 1  
Old 05-21-2009
Need help with finding unique string in log file

Shell script help
Here is 3 sample lines from a log file

<date> INFO <java.com.blah> abcd:ID= user login
<date> DEBUG <java.com.blah> <nlah bla> abcd:ID=123 user login
<date> INFO <java.com.blah> abcd:ID=3243 user login

I want to find unique "ID" from this log file which has thousands of rows. Problem is that "ID" is not in the same delimiter field and also as null sometimes "ID=" without any value which I like to ignore .

Any pointers or even possible with simple grep/awk command if not script?

Image
# 2  
Old 05-21-2009
Try this to get you going:

Code:
matt@matt-macbook$ cat test.f
ID=10
ID=10
ID=1
ID=1
ID=3
ID=4
ID=3
ID=3
ID=3
ID=
ID=3
ID=4
matt@matt-macbook$ grep -e ID=[0-9] test.f | sort | uniq
ID=1
ID=10
ID=3
ID=4
matt@matt-macbook$

# 3  
Old 05-21-2009
Code:
awk -F":" '{gsub(/user login.*/,"",$2);$1=""}1' file

# 4  
Old 05-22-2009
Code:
while(<DATA>){
	print $1,"\n" if /ID=([^ ]+)/;
}
__DATA__
<date> INFO <java.com.blah> abcd:ID= user login
<date> DEBUG <java.com.blah> <nlah bla> abcd:ID=123 user login
<date> INFO <java.com.blah> abcd:ID=3243 user login

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replace all string matches in file with unique random number

Hello Take this file... Test01 Ref test Version 01 Test02 Ref test Version 02 Test66 Ref test Version 66 Test99 Ref test Version 99 I want to substitute every occurrence of Test{2} with a unique random number, so for example, if I was using sed, substitution would be something... (1 Reply)
Discussion started by: funkman
1 Replies

2. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

3. Shell Programming and Scripting

Finding unique values in a hash (Perl)

Hi, I have a hash with unique keys associated with some data. my %FINALcontigs = ( 'mira_rep_c765:119reads**', 'ctctactggaagactgac', 'mira_rep_c7454:54reads**', 'atggatactgcgctgttgctaactactgga', 'mira_rep_c6803:12reads**', 'atcgactggatgcagggttgtggtttcta', ... (2 Replies)
Discussion started by: jdilts
2 Replies

4. Shell Programming and Scripting

Finding a string in a file

Hello All, I have a text file, i want to search for a string in it and the string is repeated multiple times in a file i want to get the first occurence of the string in a variable. the content of file is like: I want to grepthe first occurance of "Configuration flow done" and store the... (7 Replies)
Discussion started by: anand2308
7 Replies

5. Shell Programming and Scripting

Finding the number of unique words in a file

find the number of unique words in a file using sort com- mand. (7 Replies)
Discussion started by: abhikamune
7 Replies

6. Shell Programming and Scripting

Finding unique entries without sorting

Hi Guys, I have two files that I am using: File1 is as follows: wwe khfgv jfo jhgfd hoaha hao lkahe This is like a master file which has entries in the order which I want. (4 Replies)
Discussion started by: npatwardhan
4 Replies

7. Shell Programming and Scripting

get part of file with unique & non-unique string

I have an archive file that holds a batch of statements. I would like to be able to extract a certain statement based on the unique customer # (ie. 123456). The end for each statement is noted by "ENDSTM". I can find the line number for the beginning of the statement section with sed. ... (5 Replies)
Discussion started by: andrewsc
5 Replies

8. UNIX for Dummies Questions & Answers

Finding Unique strings which match pattern

I need to grep for a pattern in a file. Files are huge and have several repeated occurances of the strings which match pattern. I just need the strings which contain the pattern in the output. For eg. The contents of my file are as follows. The pattern I want to match by is ABCD ... (5 Replies)
Discussion started by: tektips
5 Replies

9. UNIX for Advanced & Expert Users

finding a string in a file

hi all.. I dont know how to search for a string in a file.. I have tried doing.. I did google but didnt get effective answers..my code is as follows: int search(char* filename,const char* username,const char* passwd) { int flag=0; unsigned long fsize=0; unsigned long current=0;... (2 Replies)
Discussion started by: Ume1986
2 Replies

10. Shell Programming and Scripting

Finding unique reocrds at a particular field

I have a pipe delimited flat file. I want to grep the records that are unique in the 4th field and repeat only once in the file for e.g.. if the file contains this 3 records i want to get the o/p as: I just gave a sample here and the file is huge one and i cant just grep from the... (7 Replies)
Discussion started by: dsravan
7 Replies
Login or Register to Ask a Question

Featured Tech Videos