Unique entries in multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Unique entries in multiple files
# 1  
Old 09-01-2015
Unique entries in multiple files

Hello,

I have a directory with a log files(many of them). Lines look like this:

Code:
Sep  1 00:05:05 server9 pop3d-ssl: LOGIN, user=abc@example.com, ip=[xxx], port=[63030]
Sep  1 00:05:05 server9 pop3d-ssl: LOGOUT, user=abc@example.com, ip=[xxx], port=[63030], top=0, retr=0, rcvd=12, sent=46, time=0
Sep  1 00:05:05 server9 imapd-ssl: couriertls: connect: Connection reset by peer
Sep  1 00:05:06 server9 pop3d: LOGIN, user=def@example.com, ip=[xxx], port=[35312]
Sep  1 00:05:06 server9 pop3d-ssl: LOGOUT, user=ghi@example.com, ip=[xxx], port=[45887], top=0, retr=0, rcvd=18, sent=238125, time=2
Sep  1 00:05:06 server9 pop3d-ssl: LOGOUT, user=jkl@example.com, ip=[xxx], port=[20521], top=0, retr=0, rcvd=12, sent=39, time=151
Sep  1 00:05:06 server9 pop3d: LOGOUT, user=def@example.com, ip=[xxx], port=[35312], top=0, retr=0, rcvd=24, sent=424, time=0
Sep  1 00:05:07 server9 pop3d-ssl: LOGIN, user=mno@example.com, ip=[xx], port=[50097]
Sep  1 00:05:07 server9 pop3d-ssl: LOGOUT, user=mno@example.com, ip=[xxx], port=[50097], top=0, retr=0, rcvd=29, sent=102, time=0

I need a script that will count unique users that login in, in theese files. E.g. if user "abc@example.com" has detected in file one, there is no need for looking for him in the rest of the files. Could you help me?
# 2  
Old 09-01-2015
Replace file.log with your log file names

Code:
$ sed -n "s/.*user=\([^,]*\),.*/\1/p" file.log | sort -u | wc -l
5

# 3  
Old 09-01-2015
Hello ramirez987,

Following may help you in same.
Code:
 awk '{match($0,/LOGIN, user=.*com/);A[substr($0,RSTART+12,RLENGTH-12)]} END{for(i in A){if(i){print i}}}'  Input_file

Output will be as follows.
EDIT: To get only the count for login ids, following may help you.
Code:
 awk '{match($0,/LOGIN, user=.*com/);if(substr($0,RSTART+12,RLENGTH-12)){o++}} END{print o}' Input_file


Thanks,
R. Singh

Last edited by RavinderSingh13; 09-01-2015 at 08:17 AM..
# 4  
Old 09-01-2015
Thank you for help.

anbu23, your solution is working, but what if I had many files in this directory and I want to do this for all of them? Lets say that I am trying to find out how many users were logged in, in e.g last week, so I will have to check 7 files, but if abc@example.com was logged yesterday and two days before, then it should count as one...

RavinderSingh13, your code looks elegent, but nothing is happend.
# 5  
Old 09-01-2015
Quote:
Originally Posted by ramirez987
Thank you for help.

anbu23, your solution is working, but what if I had many files in this directory and I want to do this for all of them? Lets say that I am trying to find out how many users were logged in, in e.g last week, so I will have to check 7 files, but if abc@example.com was logged yesterday and two days before, then it should count as one...

[...]
Hello ramirez,

Code:
sed -n "s/.*user=\([^,]*\),.*/\1/p" file.log | sort -u | wc -l

That code will still do the right thing thanks to sort -u. Repeated logins will be count as one entry.

If you have file.log, file1.log, file2.log, ..., file6.log
Code:
sed -n "s/.*user=\([^,]*\),.*/\1/p" file.log file1.log file2.log file3.log file4.log file5.log file6.log | sort -u | wc -l

or you can make use of some kind of glob
Code:
sed -n "s/.*user=\([^,]*\),.*/\1/p" file.log file[1-6].log| sort -u | wc -l


Last edited by Aia; 09-01-2015 at 10:28 AM..
# 6  
Old 09-01-2015
Thanks, but I was thinking more about some loop:

Code:
#!/bin/bash
DIR=/usr/src/count_test
FILES=$DIR/*
for i in $FILES
    do
    sed -n "s/.*user=\([^,]*\),.*/\1/p" $FILES | sort -u $i
done

But it doesn't work
# 7  
Old 09-01-2015
Quote:
Originally Posted by ramirez987
But it doesn't work
Could you explain how it doesn't work? I know why it doesn't, but I am quite sure is not the reason why you think it doesn't.

Perhaps, could you say why this, by itself, would not help?

Code:
DIR=/usr/src/count_test
FILES=$DIR/*
sed -n "s/.*user=\([^,]*\),.*/\1/p" $FILES | sort -u

This User Gave Thanks to Aia For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split into multiple files by using Unique columns in a UNIX file

I have requirement to split below file (sample.csv) into multiple files by using the unique columns (first 3 are unique columns) sample.csv 123|22|56789|ABCDEF|12AB34|2019-07-10|2019-07-10|443.3400|1|1 123|12|5679|BCDEFG|34CD56|2019-07-10|2019-07-10|896.7200|1|2... (3 Replies)
Discussion started by: RVSP
3 Replies

2. Shell Programming and Scripting

Getting unique sequences from multiple fasta file

Hi, I have a fasta file with multiple sequences. How can i get only unique sequences from the file. For example my_file.fasta >seq1 TCTCAAAGAAAGCTGTGCTGCATACTGTACAAAACTTTGTCTGGAGAGATGGAGAATCTCATTGACTTTACAGGTGTGGACGGTCTTCAGAGATGGCTCAAGCTAACATTCCCTGACACACCTATAGGGAAAGAGCTAAC >seq2... (3 Replies)
Discussion started by: Ibk
3 Replies

3. Shell Programming and Scripting

Sort unique by multiple fields

i need to sort to get all the unique records based on the 1st and 2nd column, and keep the record with the highest value on 5th column if there are duplicates, every column with varies length a^2^x^y^z bxc^2xx2^aa^bvxxxx^cdd a^3^1^2^3 a^2^x^1^c I want a result which will only keep the 1st... (2 Replies)
Discussion started by: dtdt
2 Replies

4. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

5. Shell Programming and Scripting

Reading multiple values from multiple lines and columns and setting them to unique variables.

Hello, I would like to ask for help with csh script. An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that... (7 Replies)
Discussion started by: FMMOLA
7 Replies

6. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

7. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

8. Shell Programming and Scripting

Deriving unique entries from multiple repeating patterns

Dear All, I have a below one column data.(example) Col1 1 2 . . 25 8 9 25 1 2 . . 25 Where each entry(row) is a number from 1-25, but in place whereever mentioned with . we have all the entries 1-25, but some places where ever no . like in 8 9 25 I have only 3 entries. No I... (14 Replies)
Discussion started by: ks_reddy
14 Replies

9. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

10. Shell Programming and Scripting

adding a column of unique string to multiple files

Hi, I need to add a column of unique string to individual csv file. my individual csv file looks like this and each file has variable row length: 178.52,39.4,train,0.003355544375334351,39.15752753933254,0.4895933968953914... (7 Replies)
Discussion started by: ida1215
7 Replies
Login or Register to Ask a Question