identifying duplicates line & reporting their line number - UNIX for Dummies Questions & Answers

stresslog · #1 04-23-2006

I need to find to find duplicate lines in a document and then print the line numbers of the duplicates
The files contain multiple lines with about 100 numbers on each line I need something that will output the line numbers where duplicates were found ie 1=5=7, 2=34=76

Any suggestions would be greatly appreciated - thanks for your time

stresslog · #2 04-23-2006

Hi I should clarify - I'm a post doc who is trying to teach myself unix in order to get my work done
So there is no class, this is not homework and I have no it help person to ask
I use awk but aside from spliting the file into a series of one line files and then comparing (which seems very low tech) I don't know how to do this
I don't want to delete or count the lines (using uniq) I need to find out which lines match
I'm happy to use perl or any other scripting language
I am seriously looking for suggestions - otherwise I would not have posted this !

vgersh99 · #3 04-23-2006

pls post a sample file and a desired result based on the sample.

stresslog · #4 04-23-2006

Ok here's some example input (the real data will have 100 data fields instead of ten). The first field contains the line number

1 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
7 22 35 4 85 43 4 18 39 0 0
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57

My desired output is something along the lines of
1=7
or
1 22 35 4 85 43 4 18 39 0 0 7 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57

zazzybob · #5 04-23-2006

OK, there is *no doubt* an easier/more efficient way of doing this (awk...) but...

Code:

$ cat ./inputfile
1 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
7 22 35 4 85 43 4 18 39 0 0
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57
11 22 35 4 85 43 4 18 39 0 0
12 8 6 7 70 42 7 7 3 47 53
13 5 5 8 63 48 6 4 4 50 57
$ cat ./grepit.sh
#!/bin/bash

while read line; do
  res=$(grep -n "^[0-9][0-9]*`echo ${line} | sed 's/^[0-9][0-9]*//'`$" inputfile)
  linecount=$(echo "${res}" | wc -l)
  if [ "${linecount}" -gt "1" ]; then
     echo "${res}" | awk 'BEGIN{FS=":";ORS="="} {print $1}' | sed 's/.$//'
     echo
  fi
done < inputfile | sort -nu

exit 0
$ ./grepit.sh
1=7=11
6=12
10=13

Cheers
ZB

stresslog · #6 04-23-2006

wonderful thanks so much

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Adding a columnfrom a specifit line number to a specific line number	Ezy	Shell Programming and Scripting	2	05-12-2008 05:29 AM
Appending line number to each line and getting total number of lines	chiru_h	Shell Programming and Scripting	2	03-25-2008 07:19 AM
extracting a line based on line number	narendra.pant	Shell Programming and Scripting	2	09-20-2007 02:00 AM
Appending the line number and a seperator to each line of a file ?	pjcwhite	Shell Programming and Scripting	4	03-20-2007 10:29 PM
Unix Script with line number at beginning of each line.	mascorro	Shell Programming and Scripting	5	06-19-2006 01:34 PM

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode