Printing out duplicate lines in a file with Csh


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Printing out duplicate lines in a file with Csh
# 1  
Old 03-09-2012
Printing out duplicate lines in a file with Csh

Hi guys,

I was wondering if there was an easy solution, using tcsh, to print out lines that appear twice with a given pattern in a file?

So if I am looking for lines with "test" in a given file that contains:

blah
test blah1
blah //don't print this out as it doesn't have "test" pattern
test blah2
test blah2 //this line appears twice so print this out
test blah4

If I am searching with lines that have "test" in them, I want the script to output:
test blah2

So far I am using grep to find all lines that have the pattern and outputted to a temp file. Then I am using a grep -c on the temp file to see how many of these matched pattern lines exist. If more than 2, it will print the line out. I am close to having this work but I feel like this would take a lot of time and, instead, there is a much simpler and elegant solution.

I feel like I can achieve a solution using awk. I was looking through your forums and found this awk command that counts the occurrences of a pattern.
Code:
 
awk '/,5,/ { count++ } END { print count }' test.txt

Can I extend this further such that it prints out the line with more than two matches? Appreciate any help!
# 2  
Old 03-09-2012
Your example doesn't seem to match your requirement. I'll match the example:
if test appears in the line, look for duplicates (2 only, not 3)

Code:
awk ' /test/ {arr[$0]++} 
       END{ for(i in arr) {if(arr[i]==2){print i}}}' infile > outfile

This User Gave Thanks to jim mcnamara For This Post:
# 3  
Old 03-09-2012
Thanks jim, you're amazing. That's what I was looking for. I'm guessing I should just do
Code:
 
awk ' /test/ {arr[$0]++} 
       END{ for(i in arr) {if(arr[i]>1){print i}}}' infile > outfile

if I want more than just 1 match right (so there can be three, four, etc matches)?

I've been coding away at this for a few hours now and yet you gave me that line in a few minutes. Anyway, thanks!
# 4  
Old 03-09-2012
Your guess is correct, > 1 finds all repeats.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

2. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance! (4 Replies)
Discussion started by: vlm
4 Replies

3. UNIX for Advanced & Expert Users

Inserting duplicate lines in a file

Hi, I copied the contents of a binary file into a .text file using hd (hexdump) command. The data in binary file is such that I get in many places like following 00000250 00 00 00 00 3f 2d 91 68 3f 69 fb e7 00 00 00 00 |....?-.h?i......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00... (2 Replies)
Discussion started by: KidD312
2 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. Shell Programming and Scripting

Duplicate lines in a file

Hi All, I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has: 00:44,37,67,56,15,12 00:44,34,67,56,15,12 00:44,58,67,56,15,12 00:44,35,67,56,15,12 00:59,37,67,56,15,12 00:59,34,67,56,15,12 00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies

6. Shell Programming and Scripting

CSH: step on the lines in a file

Hi, does somebody know how can I step on the lines in a file? I would like to read a file line by line. And after every line I'd like to do other commands. Lets help me! ;) (4 Replies)
Discussion started by: mig8
4 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

8. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

9. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

10. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
Login or Register to Ask a Question