identifying duplicates line & reporting their line number


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers identifying duplicates line & reporting their line number
# 1  
Old 04-23-2006
Question identifying duplicates line & reporting their line number

I need to find to find duplicate lines in a document and then print the line numbers of the duplicates
The files contain multiple lines with about 100 numbers on each line I need something that will output the line numbers where duplicates were found ie 1=5=7, 2=34=76

Any suggestions would be greatly appreciated - thanks for your time
# 2  
Old 04-23-2006
Clarification

Hi I should clarify - I'm a post doc who is trying to teach myself unix in order to get my work done
So there is no class, this is not homework and I have no it help person to ask
I use awk but aside from spliting the file into a series of one line files and then comparing (which seems very low tech) I don't know how to do this
I don't want to delete or count the lines (using uniq) I need to find out which lines match
I'm happy to use perl or any other scripting language
I am seriously looking for suggestions - otherwise I would not have posted this !
# 3  
Old 04-23-2006
pls post a sample file and a desired result based on the sample.
# 4  
Old 04-23-2006
Ok here's some example input (the real data will have 100 data fields instead of ten). The first field contains the line number

1 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
7 22 35 4 85 43 4 18 39 0 0
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57

My desired output is something along the lines of
1=7
or
1 22 35 4 85 43 4 18 39 0 0 7 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57
# 5  
Old 04-23-2006
OK, there is *no doubt* an easier/more efficient way of doing this (awk...) but...
Code:
$ cat ./inputfile
1 22 35 4 85 43 4 18 39 0 0
2 55 3 6 67 56 4 3 56 0 0
3 6 2 5 68 51 7 3 3 51 54
4 2 4 8 56 53 4 6 4 54 59
5 5 6 9 62 53 7 9 4 46 49
6 8 6 7 70 42 7 7 3 47 53
7 22 35 4 85 43 4 18 39 0 0
8 7 8 12 50 50 10 8 11 47 47
9 6 7 8 66 41 8 9 10 47 48
10 5 5 8 63 48 6 4 4 50 57
11 22 35 4 85 43 4 18 39 0 0
12 8 6 7 70 42 7 7 3 47 53
13 5 5 8 63 48 6 4 4 50 57
$ cat ./grepit.sh
#!/bin/bash

while read line; do
  res=$(grep -n "^[0-9][0-9]*`echo ${line} | sed 's/^[0-9][0-9]*//'`$" inputfile)
  linecount=$(echo "${res}" | wc -l)
  if [ "${linecount}" -gt "1" ]; then
     echo "${res}" | awk 'BEGIN{FS=":";ORS="="} {print $1}' | sed 's/.$//'
     echo
  fi
done < inputfile | sort -nu

exit 0
$ ./grepit.sh
1=7=11
6=12
10=13

Cheers
ZB
# 6  
Old 04-24-2006
Bug

wonderful thanks so much
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Identifying a sentence and putting it on a new line

I am revisiting the problem of sentence splitting. I have a Perl Script which splits a para into sentences, but acronyms and short forms create an issue #!/usr/bin/perl use feature qw/say/; use strict; use warnings; my $s; my @arr; while(<>) { chomp $_; $s .= $_ . " "; } @arr... (2 Replies)
Discussion started by: gimley
2 Replies

2. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

3. Shell Programming and Scripting

sed command to replace a line at a specific line number with some other line

my requirement is, consider a file output cat output blah sdjfhjkd jsdfhjksdh sdfs 23423 sdfsdf sdf"sdfsdf"sdfsdf"""""dsf hellow there this doesnt look good et cetc etc etcetera i want to replace a line of line number 4 ("this doesnt look good") with some other line ... (3 Replies)
Discussion started by: vivek d r
3 Replies

4. Shell Programming and Scripting

Write $line number into textfile and read from line number

Hello everyone, I don't really know anything about scripting, but I have to manage to make this script, out of necessity. #!/bin/bash while read -r line; do #I'm reading from a big wordlist instructions using $line done Is there a way to automatically write the $line number the script... (4 Replies)
Discussion started by: bobylapointe
4 Replies

5. Shell Programming and Scripting

How to use command tail -f & show line number.

Hello Guys, I have created function which is as follow: tail -f filename |grep "Key word" output from this command 19-11-2011 21:09:15,234 - INFO Numbement - error number:result = :11 19-11-2011 21:09:15,286 - INFO Numbement - error number:result = :11 19-11-2011 21:09:15,523 - INFO... (5 Replies)
Discussion started by: ooilinlove
5 Replies

6. UNIX for Dummies Questions & Answers

Identifying the first line that has zeros

If I have a file like: 9350. 0.288426 9370. 0.320469 9390. 0.394475 9410. 0.353157 9430. 0.336001 9450. 0.336692 9470. 0.356827 9490. 0.359891 9510. 0.346305 9530. 0.356506 9550. 0.348306 9570. 0.36832 9590. 0.379067 9610. 0.0246704 9630. 0 9650. 0 9670. 0 (5 Replies)
Discussion started by: cosmologist
5 Replies

7. UNIX for Dummies Questions & Answers

How to read contents of a file from a given line number upto line number again specified by user

Hello Everyone. I am trying to display contains of a file from a specific line to a specific line(let say, from line number 3 to line number 5). For this I got the shell script as shown below: if ; then if ; then tail +$1 $3 | head -n $2 else ... (5 Replies)
Discussion started by: grc
5 Replies

8. Shell Programming and Scripting

how to get the data from line number 1 to line number 100 of a file

Hi Everybody, I am trying to write a script that will get some perticuler data from a file and redirect to a file. My Question is, I have a Very huge file,In that file I have my required data is started from 25th line and it will ends in 100th line. I know the line numbers, I need to get all... (9 Replies)
Discussion started by: Anji
9 Replies

9. Shell Programming and Scripting

Adding a columnfrom a specifit line number to a specific line number

Hi, I have a huge file & I want to add a specific text in column. But I want to add this text from a specific line number to a specific line number & another text in to another range of line numbers. To be more specific: lets say my file has 1000 lines & 4 Columns. I want to add text "Hello"... (2 Replies)
Discussion started by: Ezy
2 Replies

10. Shell Programming and Scripting

Appending line number to each line and getting total number of lines

Hello, I need help in appending the line number of each line to the file and also to get the total number of lines. Can somebody please help me. I have a file say: abc def ccc ddd ffff The output should be: Instance1=abc Instance2=def Instance3=ccc Instance4=ddd Instance5=ffff ... (2 Replies)
Discussion started by: chiru_h
2 Replies
Login or Register to Ask a Question