how can I delete duplicates in the log?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting how can I delete duplicates in the log?
# 1  
Old 09-02-2009
how can I delete duplicates in the log?

I have a log file and I am trying to run a script against it to search for key issues such as invalid users, errors etc. In one part, I grep for session closed and get a lot of the same thing,, ie. root username etc. I want to remove the multiple root and just have it do a count, like wc -l

These are the unidentified users
auth could not identify password for [henrys]: 1 Time(s)
auth could not identify password for [henrys]: 1 Time(s)
auth could not identify password for [henrys]: 2 Time(s)
auth could not identify password for [henrys]: 1 Time(s)
Maybe a simple variable tip?
or a for i loop?
Thanks, just learning scripting.
# 2  
Old 09-02-2009
if the entries are exactly similar.
"cat logfile | sort -u "will work fine

in your case:
"cat logfile| awk -F':' '{print $1}'| sort -u" should work
# 3  
Old 09-02-2009
Quote:
Originally Posted by vickylife
if the entries are exactly similar.
"cat logfile | sort -u "will work fine

in your case:
"cat logfile| awk -F':' '{print $1}'| sort -u" should work
why exactly do you need to cat a file?
# 4  
Old 09-02-2009
sort -u worked great. Thanks so much!

---------- Post updated at 03:09 PM ---------- Previous update was at 03:02 PM ----------

why do I need to cat the file? I am trying to format a log so that the problems come out such as errors, invalid users, illegal, authentication failures etc. are in the report. I would like to clean it up as well, so that you don't have 100 entries if a single user had issues. Ideally it would do a count first, then may be state the user tried to login 10 times for e.g. Thanks V.
# 5  
Old 09-02-2009
Quote:
Originally Posted by taekwondo
sort -u worked great. Thanks so much!

---------- Post updated at 03:09 PM ---------- Previous update was at 03:02 PM ----------

why do I need to cat the file? I am trying to format a log so that the problems come out such as errors, invalid users, illegal, authentication failures etc. are in the report. I would like to clean it up as well, so that you don't have 100 entries if a single user had issues. Ideally it would do a count first, then may be state the user tried to login 10 times for e.g. Thanks V.
Follow the posted 'cat' link.
Code:
awk -F':' '{print $1}' logfile | sort -u

# 6  
Old 09-04-2009
Thank you very much.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To Delete the duplicates using Part of File Name

I am using the below script to delete duplicate files but it is not working for directories with more than 10k files "Argument is too long" is getting for ls -t. Tried to replace ls -t with find . -type f \( -iname "*.xml" \) -printf '%T@ %p\n' | sort -rg | sed -r 's/* //' | awk... (8 Replies)
Discussion started by: gold2k8
8 Replies

2. Shell Programming and Scripting

Delete only if duplicates found in each record

Hi, i have another problem. I have been trying to solve it by myself but failed. inputfile ;; ID T08578 NAME T08578 SBASE 30696 EBASE 32083 TYPE P func just test func chronology func cholesterol func null INT 30765-37333 INT 37154-37318 Link 5546 Link 8142 (4 Replies)
Discussion started by: redse171
4 Replies

3. Shell Programming and Scripting

Delete duplicates in CA bundle

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates. The alignment of the certificate within the bundle seems to be different. Example: Cert 1... (7 Replies)
Discussion started by: diva_thilak
7 Replies

4. Shell Programming and Scripting

Search for duplicates and delete but remain the first one based on a specific pattern

Hi all, I have been trying to delete duplicates based on a certain pattern but failed to make it works. There are more than 1 pattern which are duplicated but i just want to remove 1 pattern only and remain the rest. I cannot use awk '!x++' inputfile.txt or sed '/pattern/d' or use uniq and sort... (7 Replies)
Discussion started by: redse171
7 Replies

5. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478... (5 Replies)
Discussion started by: pareshkp
5 Replies

6. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

7. Shell Programming and Scripting

Delete Duplicates on the basis of two column values.

Hi All, i need ti delete two duplicate processss which are running on the same device type (column 1) and port ID (column 2). here is the sample data p1sc1m1 15517 11325 0 01:00:24 ? 0:00 scagntclsx25octtcp 2967 in3v mvmp01 0 8000 N S 969 750@751@752@ p1sc1m1 15519 11325 0 01:00:24 ? ... (5 Replies)
Discussion started by: neeraj617
5 Replies

8. Shell Programming and Scripting

Delete duplicates via script?

Hello, i have the following problem: there are two folders with a lot of files. Example: FolderA contains AAA, BBB, CCC FolderB contains DDD, EEE, AAA How can i via script identify AAA as duplicate in Folder B and delete it there? So that only DDD and EEE remain, in Folder B? Thank you... (16 Replies)
Discussion started by: Y-T
16 Replies

9. Shell Programming and Scripting

How can i delete the duplicates based on one column of a line

I have my data something like this (08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb (08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa (08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts (08/03/2009 22:57:42.425)(:) Ravi... (11 Replies)
Discussion started by: rdhanek
11 Replies

10. Shell Programming and Scripting

An interactive way to delete duplicates

1)I am trying to write a script that works interactively lists duplicated records on certain field/column and asks user to delete one or more. And finally it deletes all the records the used has asked for. I have an idea to store those line numbers in an array, not sure how to do this in... (3 Replies)
Discussion started by: chvs2000
3 Replies
Login or Register to Ask a Question