Print duplicate lines


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Print duplicate lines

I have a file where some of the lines are duplicates.

How do I use bash to print all the lines that have duplicates?
# 2  
Code:
awk 'arr[$0]++; END {for(i in arr ){ if(arr[i]>1){print i}   }   }' inputfile

# 3  
Code:
awk '$0==s{print s RS $0}{s=$0}' file

Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Print number of lines for files in directory, also print number of unique lines

I have a directory of files, I can show the number of lines in each file and order them from lowest to highest with: wc -l *|sort 15263 Image.txt 16401 reference.txt 40459 richtexteditor.txt How can I also print the number of unique lines in each file? 15263 1401 Image.txt 16401... (15 Replies)
Discussion started by: spacegoose
15 Replies

2. Shell Programming and Scripting

duplicate elimination with column print

hi, my input a b c d e f g k i m g i k l b i n g o h m a b c d e b i n g o my output a b c d e f g k i m g i k l b i n g o h m what I would like to do is compare the first five columns of each row to the entire records in the file and if there is a match, eliminate that... (1 Reply)
Discussion started by: jacobs.smith
1 Replies

3. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

4. Shell Programming and Scripting

Awk: How to merge duplicate lines and print in a single

The input file: >cat module1 200611051053 95 200523457498 35 200617890187 57 200726098123 66 200645676712 71 200744556590 68 >cat module2 200645676712 ... (10 Replies)
Discussion started by: winter9
10 Replies

5. Shell Programming and Scripting

Print duplicate only lines as normal output - Awk

input output a1 100 200 XYZ_X a1 98 188 ABC (2 Replies)
Discussion started by: quincyjones
2 Replies

6. UNIX for Dummies Questions & Answers

print duplicate number

input chr1 10 100 chr1 10 100 chr1 20 200 output chr1 10 100 2 chr1 20 200 1 (1 Reply)
Discussion started by: repinementer
1 Replies

7. Shell Programming and Scripting

print first few lines, then apply regex on a specific column to print results.

abc.dat tty cpu tin tout us sy wt id 0 0 7 3 19 71 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 133.2 0.0 682.9 0.0 1.0 0.0 7.2 0 79 c1t0d0 0.2 180.4 0.1 5471.2 3.0 2.8 16.4 15.6 15 52 aaaaaa1-xx I want to skip first 5 line... (4 Replies)
Discussion started by: kchinnam
4 Replies

8. Shell Programming and Scripting

print lines AFTER lines cointaining a regexp (or print every first and fourth line)

Hi all, This should be very easy but I can't figure it out... I have a file that looks like this: @SRR057408.1 FW8Y5CK02R652T length=34 AGCAGTGGTATCAACGCAGAGTAAGCAGTGGTAT +SRR057408.1 FW8Y5CK02R652T length=34 FIIHFF6666?=:88@@@BBD:::?@ABBAAA>8 @SRR057408.2 FW8Y5CK02TBMHV length=52... (1 Reply)
Discussion started by: kmkocot
1 Replies

9. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

10. Shell Programming and Scripting

Print Only second Duplicate entry in the file

I have file where it contains 2 columns. In two columns the first column is repeated more than once. I wanted to take the unique record in first column and the corresponding second column value . The below is the example of the file: 8244100320012955|000b063471a4... (4 Replies)
Discussion started by: ravi_rn
4 Replies

Featured Tech Videos