05-26-2010
How to find duplicate line in log file?
Hi guys, I'm really happy to find this forum
I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated?
I already did the grep, and sort the file? now I don't know how can I find the duplicated lines and counter for that?
any help or idea
thanks a lot.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi all
pls help me by providing soln for my problem
I'm having a text file which contains duplicate records .
Example:
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies
2. Shell Programming and Scripting
Thanks
AVKlinux (6 Replies)
Discussion started by: avklinux
6 Replies
3. Shell Programming and Scripting
Dear All,
I have one file which looks like :
account1:passwd1
account2:passwd2
account3:passwd3
account1:passwd4
account5:passwd5
account6:passwd6
you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies
4. Shell Programming and Scripting
I have somelogfile.log where I'm trying to find only the numbers which would start at c28 on the third line from the bottom and end at the next comma, so for somelogfile.log:
bunch of text - 0, some other text - 123
or
bunch of text - 123, some other text - 192831923
where I want either the... (5 Replies)
Discussion started by: unclecameron
5 Replies
5. Shell Programming and Scripting
Hi,
Need to find a duplicate records on the first column,
ANU4501710430989 0000000W20389390
ANU4501710430989 0000000W67065483
ANU4501130050520 0000000W80838713
ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies
6. UNIX for Dummies Questions & Answers
Hi Unix gurus,
Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me.
File format: CSV file
File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies
7. Shell Programming and Scripting
Hi,
In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'.
In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies
8. Shell Programming and Scripting
Hi, Gurus,
I need find the duplicate record in unix file.
what command I should use for this.
Thanks in advance (4 Replies)
Discussion started by: ken6503
4 Replies
9. Shell Programming and Scripting
I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code:
awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies
10. UNIX for Dummies Questions & Answers
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
look(1) General Commands Manual look(1)
NAME
look - Finds lines in a sorted list
SYNOPSIS
look [-df] [-tcharacter] string [file]
The look command prints all lines in a sorted file that begin with string.
OPTIONS
Uses dictionary order; only letters, digits, tabs, and spaces are used in comparisons. Searches without regard to case; treats uppercase
and lowercase as equivalent. Ignores character and characters following it in the search string. If you specify look -tC ABCDE, the
string ABCDE would become (in effect) AB, with CDE being ignored. This option is primarily for shell scripts, in which more than one
string is being processed.
DESCRIPTION
If no file is specified, look searches in the system word list /usr/share/dict/words, with the options -df assumed by default.
The look command uses binary search.
The -d and -f options affect comparisons as in sort.
NOTES
In order to use the -f option, you must first sort file with the sort -f command; otherwise, look displays only lowercase items.
If you do not specify -f, but specify a file (such as /usr/share/dict/words) that has been sorted with sort -f, look may not produce any
output.
EXAMPLES
To search a sorted file called sortfile for all lines that begin with the string as, enter: look as sortfile To search the system word list
for all words beginning with smi, enter: look smi
This might result in: smile smirk smith smithereens Smithfield Smithson smithy smitten
FILES
System word list.
SEE ALSO
Commands: grep(1), sort(1), spell(1)
look(1)