Find duplicates in file with line numbers


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Find duplicates in file with line numbers
# 1  
Old 07-14-2018
Find duplicates in file with line numbers

Hello All,

This is a noob question. I tried searching for the answer but the answer found did not help me .

I have a file that can have duplicates.
Code:
100
200
300
400
100
150

the number 100 is duplicated twice. I want to find the duplicate along with the line number.

expected output
Code:
100 5

please help

Regards
David


Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 07-14-2018 at 05:59 PM.. Reason: Added CODE tags.
# 2  
Old 07-14-2018
Welcome to the forum.


Sure none of the links given at bottom left (under "More UNIX and Linux Forum Topics You Might Find Helpful") can't help? e.g. this one?
# 3  
Old 07-14-2018
I did look at those links.
I tried
Code:
awk -F: 'x[$1]++ { print $1 " is duplicated"}' fileName

The output I get is
Code:
is duplicated

I want the duplicate value along with the line number
# 4  
Old 07-15-2018
That seems an indicator your file has non-*nix DOS line terminators (<carriage return>, <CR>, \r, ^M, 0x0D). Remove before continuing, e.g. with dos2unix.


There's no colon in the file - no reason to set it as FS. Try
Code:
awk 'a[$1]++ {print $0, NR}' file
100 5

This User Gave Thanks to RudiC For This Post:
# 5  
Old 07-15-2018
If you don't want to use dos2unix before processing your input files, you can also build that code into your awk script:
Code:
awk '
{	sub(/\r$/, "")
}
a[$1]++ {
	print $0, NR
}' file

Note that the above code prints complete lines when the 1st field on the given lines is duplicated. If you want to only print the 1st field when the 1st field is duplicated, use:
Code:
awk '
{	sub(/\r$/, "")
}
a[$1]++ {
	print $1, NR
}' file

If you want to print complete lines when complete lines are duplicated, use:
Code:
awk '
{	sub(/\r$/, "")
}
a[$0]++ {
	print $0, NR
}' file

This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find Special/Null/Control Chars and Print Line Numbers

Hi All, This might be a basic question... I need to write a script to find all/any Speacial/Null/Control Chars and Print Line Numbers from an input file. Output something like Null Characters in File Name at : Line Numbers Line = Print the line Control Characters in File Name at : Line... (2 Replies)
Discussion started by: Kevin Tivoli
2 Replies

2. Shell Programming and Scripting

How to find numbers in text file?

Hi I have a text file with rows like this: 7 Herman ASI-40 Jungle (L) Blueprint (L) Weapon Herman ASI-40 Jungle (L) 215.00 57 65.21 114.41 and 9 Herman CAP-505 (L) Blueprint (L) Weapon Herman CAP-505 (L) 220.00 46.84 49.1 104.82 and 2 ClericDagger 1C blueprint Melee - Shortblade... (2 Replies)
Discussion started by: pesa
2 Replies

3. Shell Programming and Scripting

find all numbers > x and replace with y within a file

How would I do this? How could i use <> symbols for numbers in the find/replace code below? perl -pi -e 's/test/tst/' OR is there a better way? 100 5000 2 432 4 2 33 4 5 6 65 300 301 needs to be: 100 300 2 300 4 2 33 4 5 6 65 300 300 also it might not always need spaces... i... (12 Replies)
Discussion started by: herot
12 Replies

4. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

5. Shell Programming and Scripting

Assign Line Numbers to each line of the file

Hi! I'm trying to assign line numbers to each line of the file for example consider the following.. The contents of the input file are hello how are you? I'm fine. How about you? I'm trying to get the following output.. 1 hello how are you? 2 I'm fine. 3 How about you? ... (8 Replies)
Discussion started by: abk07
8 Replies

6. Shell Programming and Scripting

Find duplicates in the first column of text file

Hello, My text file has input of the form abc dft45.xml ert rt653.xml abc ert57.xml I need to write a perl script/shell script to find duplicates in the first column and write it into a text file of the form... abc dft45.xml abc ert57.xml Can some one help me plz? (5 Replies)
Discussion started by: gameboy87
5 Replies

7. Shell Programming and Scripting

Unix help to find blank lines in a file and print numbers on that line

Hi, I would like to know how to solve one of my problems using expert unix commands. I have a file with occasional blank lines; for example; dertu frthu fghtu frtty frtgy frgtui frgtu ghrye frhutp frjuf I need to edit the file so that the file looks like this; (10 Replies)
Discussion started by: Lucky Ali
10 Replies

8. UNIX for Dummies Questions & Answers

Adding Numbers in a line from a file

Hello everyone ! Ive searched everywhere and still havnt found enough information to help me overcome (what seems like) a small problem I have created a temporary file in which i store numbers which a seperated by a space, eg) 5 10 46 23 866 392 i wish to take the numbers for each line... (2 Replies)
Discussion started by: healwithchaos
2 Replies

9. UNIX for Dummies Questions & Answers

Question About Getting Line Numbers in a File

Ok, this is a unique question. Say I have a word like "jamamamama" in a file called foo.bar. Now, how do you get the line number that the word "jamammama" exist in the file foo.bar without having to go into the foo.bar file to edit it? is there a command i can run on the foo.bar... (4 Replies)
Discussion started by: SkySmart
4 Replies

10. Shell Programming and Scripting

find 2 line numbers, grab text in between

hi, i have a large text file that I just want to extract the important information from. It will be a random number of lines but between two specific line numbers/markers. I was thinking I could get the line number for the first marker: Tablespace Percent Total Free Then get the line... (11 Replies)
Discussion started by: Da_Duck
11 Replies
Login or Register to Ask a Question