Sponsored Content
Top Forums Shell Programming and Scripting identifying duplicate entries Post 302431039 by cesarNZ on Sunday 20th of June 2010 07:10:53 PM
Old 06-20-2010
identifying duplicate entries

hi all,
have got a large log file and was wondering if there is a easy way on solaris box to grep out duplicate entries based on email address ??

sample log file :

Code:
2010-06-19,04:08:12,235632470,2010-06-18T00:00:00.000+12:00,zinny123@hotmail.com
2010-06-19,04:09:57,235632470,2010-06-18T00:00:00.000+12:00,zinny123@hotmail.com
2010-06-19,04:28:36,223906214,2010-06-18T00:00:00.000+12:00,zkml123@xtra.co.nz
2010-06-19,04:01:51,101427641,2010-06-18T00:00:00.000+12:00,zl2t890@orcon.net.nz
2010-06-19,04:03:40,101427641,2010-06-18T00:00:00.000+12:00,zl2890@orcon.net.nz


thanks in advance.
 

10 More Discussions You Might Find Interesting

1. Solaris

Duplicate crontab entries

hi guys. can someone tell me what will happen if there are two identical crontab entry for an application. For example 03 23 * * 1 /usr/vt/crondemo 03 23 * * 1 /usr/vt/crondemo will the file crondemo run twice, once or wont run at all?? (2 Replies)
Discussion started by: vikashtulsiyan
2 Replies

2. Shell Programming and Scripting

Removal of Duplicate Entries from the file

I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File. The example of the File is shown below: 8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7 ... (1 Reply)
Discussion started by: ravi_rn
1 Replies

3. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi All, I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like Code: wcars42g wcars89j wcars42g wcars42b wcars42b Will duplicate entries... (1 Reply)
Discussion started by: akash_mahakode
1 Replies

4. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi, I forgot how to start a new thread. :( Can somebody please guide me? I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like wcars42g... (2 Replies)
Discussion started by: akash_mahakode
2 Replies

5. Solaris

duplicate PATH entries

I noticed we have duplicate (or even triple) entries in PATH env variable. Is there any reason to have them all or is it safe to leave only one? thanks. (2 Replies)
Discussion started by: orange47
2 Replies

6. Shell Programming and Scripting

Identifying entries based on 2 fields in a string.

Hi Guys, I’m struggling to use two fields to do a duplicate/ unique by output. I want to look IP addresses assigned to more than one account during a given period in the logs. So duplicate IP and account > 1 then print all the logs for that IP. I have been Using AWK (just as its installed... (3 Replies)
Discussion started by: wabbit02
3 Replies

7. Shell Programming and Scripting

How to find duplicate entries

I have a file contails as below I/P: 123456 123456 234567 987654 678905 678905 Like above i have 1000's of entries I need output as below O/P: 123456 678905 I'm using uniq -d filename it is showing results but it is missing few duplicate entries and i dont know why.Please... (9 Replies)
Discussion started by: buzzme
9 Replies

8. Shell Programming and Scripting

Append values of duplicate entries

My input file is: LOC_Os01g01870 GO:0006139 LOC_Os01g01870 GO:0009058 LOC_Os01g02570 GO:0006464 LOC_Os01g02570 GO:0009987 LOC_Os01g02570 GO:0008152 LOC_Os01g04380 GO:0006950 LOC_Os01g04380 GO:0009628 I want to append the duplicate values in a tab/space... (2 Replies)
Discussion started by: Sanchari
2 Replies

9. Shell Programming and Scripting

How to keep the last 2 records from duplicate entries?

Gents, Please how I can get only the last 2 records from repetead values, from column 2 input 1 1011 1 1011 1 1012 1 1012 1 5001 1 5001 1 5002 1 5002 1 5003 1 5003 1 7001 1 7001 1 7002 1 7002 (2 Replies)
Discussion started by: jiam912
2 Replies

10. Shell Programming and Scripting

Remove columns with duplicate entries

I have a 13gb file. It has the following columns: The 3rd column is basically correlation values. I want to delete those rows which are repeated between the columns: A B 0.04 B C 0.56 B B 1 A A 1 C D 1 C C 1 Desired Output: (preferably in a .csv format A,B,0.04 B,C,0.56 C,D,1... (3 Replies)
Discussion started by: Sanchari
3 Replies
VARNISHTOP(1)															     VARNISHTOP(1)

NAME
varnishtop - Varnish log entry ranking SYNOPSIS
varnishtop [-1] [-b] [-C] [-c] [-d] [-f] [-I regex] [-i tag] [-n varnish_name] [-r file] [-V] [-X regex] [-x tag] DESCRIPTION
The varnishtop utility reads varnishd(1) shared memory logs and presents a continuously updated list of the most commonly occurring log entries. With suitable filtering using the -I, -i, -X and -x options, it can be used to display a ranking of requested documents, clients, user agents, or any other information which is recorded in the log. The following options are available: -1 Instead of presenting of a continuously updated display, print the statistics once and exit. Implies -d. -b Include log entries which result from communication with a backend server. If neither -b nor -c is specified, varnishtop acts as if they both were. -C Ignore case when matching regular expressions. -c Include log entries which result from communication with a client. If neither -b nor -c is specified, varnishtop acts as if they both were. -d Process old log entries on startup. Normally, varnishtop will only process entries which are written to the log after it starts. -f Sort and group only on the first field of each log entry. This is useful when displaying e.g. stataddr entries, where the first field is the client IP address. -I regex Include log entries which match the specified regular expression. If neither -I nor -i is specified, all log entries are included. -i tag Include log entries with the specified tag. If neither -I nor -i is specified, all log entries are included. -p period Specifies the number of seconds to measure over, the default is 60 seconds. The first number in the list is the average number of requests seen over this time period. -n Specifies the name of the varnishd instance to get logs from. If -n is not specified, the host name is used. -r file Read log entries from file instead of shared memory. -V Display the version number and exit. -X regex Exclude log entries which match the specified regular expression. -x tag Exclude log entries with the specified tag. EXAMPLES
The following example displays a continuously updated list of the most frequently requested URLs:: varnishtop -i RxURL The following example displays a continuously updated list of the most commonly used user agents:: varnishtop -i RxHeader -C -I ^User-Agent SEE ALSO
o varnishd(1) o varnishhist(1) o varnishlog(1) o varnishncsa(1) o varnishstat(1) HISTORY
The varnishtop utility was originally developed by Poul-Henning Kamp in cooperation with Verdens Gang AS and Varnish Software AS, and later substantially rewritten by Dag-Erling Smorgrav. This manual page was written by Dag-Erling Smorgrav. COPYRIGHT
This document is licensed under the same licence as Varnish itself. See LICENCE for details. o Copyright (c) 2006 Verdens Gang AS o Copyright (c) 2006-2011 Varnish Software AS AUTHOR
Dag-Erling Smorgrav 1.0 2010-05-31 VARNISHTOP(1)
All times are GMT -4. The time now is 12:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy