To the original poster: Carlo could give all the details, but the big difference I see between the two solutions he gave is that one grabs the first occurrence of a line, and the other grabs the last (meaning, you'll either get the first timestamp, or--probably more helpful--the timestamp of the last occurrence, depending on which you pick). The impressive perl solution also grabs the last occurrence.
Oh yeah, but your file is backwards from traditional logfiles--I forgot. Anywhoo.. I'm sure you'll work it out. Glad to read these solutions.
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.
FileA
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
and I want to simply it with no duplicate line as file... (5 Replies)
I have following file content (3 fields each line):
23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
df dag 10.0.0.5
dfd dfdas 10.0.0.5
dfd dfd 10.0.0.6
daf nfd 10.0.0.6
...
as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Hi,
I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates.
Thanks in advance,
sudvishw :confused: (7 Replies)
Hi, I have a huge file which is about 50GB. There are many lines. The file format likes
21 rs885550 0 9887804 C C T C C C C C C C
21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0
21 rs303304 0 9941889 A A A A A A A A A A
22 rs303304 0 9941890 0 A A A A A A A A A
The question is that there are a few... (4 Replies)
Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file.
I wrote the following script, but does not work. What is the problem?
The output file should only contain five lines:
Later! (5 Replies)
Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be.
Need a bash script or command that goes sequentially through all lines of a file, and does this:
if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Hi,
I have a csv file which contains some millions of lines in it.
The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line).
I don't want to use any pattern from the Header as I have some... (7 Replies)
Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings.
Eg;
One and a Two
Unix.com is the Best
This as a Line Line
Example duplicate sentence with the word... (22 Replies)
Hi All,
I am storing the result in the variable result_text using the below code.
result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines.
file and time for the interval 03:30 - 03:45
file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
LEARN ABOUT PLAN9
grep
GREP(1) General Commands Manual GREP(1)NAME
grep - search a file for a pattern
SYNOPSIS
grep [ option ... ] pattern [ file ... ]
DESCRIPTION
Grep searches the input files (standard input default) for lines (with newlines excluded) that match the pattern, a regular expression as
defined in regexp(6). Normally, each line matching the pattern is `selected', and each selected line is copied to the standard output.
The options are
-c Print only a count of matching lines.
-h Do not print file name tags (headers) with output lines.
-i Ignore alphabetic case distinctions. The implementation folds into lower case all letters in the pattern and input before interpre-
tation. Matched lines are printed in their original form.
-l (ell) Print the names of files with selected lines; don't print the lines.
-L Print the names of files with no selected lines; the converse of -l.
-n Mark each printed line with its line number counted in its file.
-s Produce no output, but return status.
-v Reverse: print lines that do not match the pattern.
Output lines are tagged by file name when there is more than one input file. (To force this tagging, include /dev/null as a file name
argument.)
Care should be taken when using the shell metacharacters $*[^|()= and newline in pattern; it is safest to enclose the entire expression in
single quotes '...'.
SOURCE
/sys/src/cmd/grep.c
SEE ALSO ed(1), awk(1), sed(1), sam(1), regexp(6)DIAGNOSTICS
Exit status is null if any lines are selected, or non-null when no lines are selected or an error occurs.
GREP(1)