Sponsored Content
Top Forums Shell Programming and Scripting Need to remove the duplicate lines from a log!! Post 302583422 by CarloM on Tuesday 20th of December 2011 11:18:45 AM
Old 12-20-2011
Code:
awk -F"TM" '{lines[$1]=$2} END {for (i in lines) {print i lines[i]}}' inputfile

or
Code:
awk -F"TM" '!(lines[$1]) {lines[$1]=$2} END {for (i in lines) {print i lines[i]}}' inputfile

Quote:
Originally Posted by ryran
If you need timestamps, I can offer this BASH snippet, but it will only get rid of duplicate ADJACENT lines
You could sort the file on input.
These 2 Users Gave Thanks to CarloM For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

3. Shell Programming and Scripting

remove duplicate lines using awk

Hi, I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates. Thanks in advance, sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies

4. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

6. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
CONTROL.CTL(5)							File Formats Manual						    CONTROL.CTL(5)

NAME
control.ctl - specify handling of Usenet control messages DESCRIPTION
The file /etc/news/control.ctl is used to determine what action is taken when a control message is received. It is read by the parsecon- trol script, which is called by all the control scripts. (For an explanation of how the control scripts are invoked, see innd(8).) The file consists of a series of lines; blank lines and lines beginning with a number sign (``#'') are ignored. All other lines consist of four fields separated by a colon: message:from:newsgroups:action The first field is the name of the message for which this line is valid. It should be either the name of the control message, or the word ``all'' to mean that it is valid for all messages. The second field is a shell-style pattern that matches the email address of the person posting the message. (The poster's address is first converted to lowercase.) The matching is done using the shell's case statement; see sh (1) for details. If the control message is ``newgroup'' or ``rmgroup'' then the third field specifies the shell-style pattern that must match the group being created or removed. If the control message is of a different type, then this field is ignored. The fourth field specifies what action to take if this line is selected for the message. The following actions are understood: doit The action requested by the control message should be performed. In most cases the control script will also send mail to usenet. doifarg If the control message has an argument, this is treated as a ``doit'' action. If no argument was given, it is treated as a ``mail'' entry. This is used in ``sendsys'' entries script so that a site can request its own newsfeeds(5) entry by posting a ``sendsys mysite'' article. On the other hand, sendsys ``bombs'' ask that the entire newsfeeds file be sent to a forged reply-to address; by using ``doifarg'' such messages will not be processed automatically. doit=file The action is performed, but a log entry is written to the specified log file, file. If file is the word ``mail'' then the record is mailed. A null string is equivalent to /dev/null. A pathname that starts with a slash is taken as the absolute filename to use as the log. All other pathnames are written to /var/log/news/file.log. The log is written by writelog (see newslog(8)). drop No action is taken; the message is ignored. log A one-line log notice is sent to standard error. Innd normally directs this to the file /var/log/news/errlog. log=file A log entry is written to the specified log file, file, which is interpreted as described above. mail A mail message is sent to the news administrator. Lines are matched in order; the last match found in the file is the one that is used. For example, with the following three lines: newgroup:*:*:drop newgroup:tale@*.uu.net:comp.*|misc.*|news.*|rec.*|sci.*|soc.*|talk.*:doit newgroup:kre@munnari.oz.au:aus.*:mail A newgroup coming from ``tale'' at a UUNET machine will be honored if it is in the mainstream Usenet hierarchy. If ``kre'' posts a new- group message creating ``aus.foo'', then mail will be sent. All other newgroup messages are ignored. HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. This is revision 1.8, dated 1996/09/06. SEE ALSO
innd(8), newsfeeds(5), scanlogs(8). CONTROL.CTL(5)
All times are GMT -4. The time now is 04:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy