Duplicate lines in a file


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Duplicate lines in a file

Hi All,

I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has:

Code:
00:44,37,67,56,15,12
00:44,34,67,56,15,12
00:44,58,67,56,15,12
00:44,35,67,56,15,12
00:59,37,67,56,15,12
00:59,34,67,56,15,12
00:59,35,67,56,15,12
00:59,58,67,56,15,12
01:14,35,68,53,15,12
01:14,37,68,53,15,12
01:14,34,68,53,15,12
01:14,58,68,53,15,12

I am trying to get the output as :
Code:
00:44,37,67,56,15,12
00:59,37,67,56,15,12
01:14,35,68,53,15,12

So basically what I am trying here is - if a line is repeated more than once, I want it to be printed(or stored in a file) just once.

Have tried using "uniq" command, but doesn't seem to be working.

Any help would be greatly appreciated. Thanks in advance!!
# 2  
Hi.

None of your lines are actually duplicated (they're all unique). You mean the first field?

Code:
$ awk -F, '!A[$1]++' inputfile
00:44,37,67,56,15,12
00:59,37,67,56,15,12
01:14,35,68,53,15,12

# 3  
Quote:
Originally Posted by scottn
Hi.

None of your lines are actually duplicated (they're all unique). You mean the first field?

Code:
$ awk -F, '!A[$1]++' inputfile
00:44,37,67,56,15,12
00:59,37,67,56,15,12
01:14,35,68,53,15,12


Hi Scottn,

Yes, I do mean the first field, sorry for missing out that earlier.

The awk command does not seem to be working for me Smilie

Code:
hws006a001: awk -F, '!A[$1]++' diskspace_Dywhapp_DR
awk: syntax error near line 1
awk: bailing out near line 1

Any idea what the error is?

Just for the record, the Unix environment is SunOS and I have tried usking ksh and bash. Thanks..
# 4  
Quote:
Originally Posted by faiz1985
Hi Scottn,

Yes, I do mean the first field, sorry for missing out that earlier.

The awk command does not seem to be working for me Smilie

Code:
hws006a001: awk -F, '!A[$1]++' diskspace_Dywhapp_DR
awk: syntax error near line 1
awk: bailing out near line 1

Any idea what the error is?

Just for the record, the Unix environment is SunOS and I have tried usking ksh and bash. Thanks..
Smilie Good ol' Solaris awk!

Use nawk, or /usr/xpg4/bin/awk
# 5  
Quote:
Originally Posted by scottn
Smilie Good ol' Solaris awk!

Use nawk, or /usr/xpg4/bin/awk
nawk worked perfectly Smilie

Thanks a lot for your help buddy Smilie
# 6  
Hi, faiz1985:

scottn's awk approach works great, but it doesn't hurt to know other possible solutions Smilie
Code:
sort -ut, -k1,1 inputfile

Cheers,

Alister
# 7  
The two solutions don't give the same result:
Scottn's solution :The first record for each key is selected.
Code:
00:44,37,67,56,15,12
00:59,37,67,56,15,12
01:14,35,68,53,15,12

alister's solution : The first record in alphabetic order for each key is selected
Code:
00:44,34,67,56,15,12
00:59,34,67,56,15,12
01:14,34,68,53,15,12

Jean-Pierre.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #69
Difficulty: Easy
Grep stands for 'global regular expression process'.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

2. UNIX for Dummies Questions & Answers

Duplicate lines in a file

I have a file with following data A B C I would like to print like this n times(For eg:5 times) A B C A B C A B C A B C A (7 Replies)
Discussion started by: nsuresh316
7 Replies

3. Shell Programming and Scripting

bash keep only duplicate lines in file

hello all in my bash script I have a file and I only want to keep the lines that appear twice in the file.Is there a way to do this? thanks in advance! (4 Replies)
Discussion started by: vlm
4 Replies

4. UNIX for Advanced & Expert Users

Inserting duplicate lines in a file

Hi, I copied the contents of a binary file into a .text file using hd (hexdump) command. The data in binary file is such that I get in many places like following 00000250 00 00 00 00 3f 2d 91 68 3f 69 fb e7 00 00 00 00 |....?-.h?i......| 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00... (2 Replies)
Discussion started by: KidD312
2 Replies

5. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

8. UNIX for Dummies Questions & Answers

How to redirect duplicate lines from a file????

Hi, I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file. Suppose I have a file called file_dup.txt which contains some line as file_dup.txt A100-R1 ACCOUNTING-CONTROL ACTONA-ACTASTOR ADMIN-AUTH-STATS ACTONA-ACTASTOR... (3 Replies)
Discussion started by: zing_foru
3 Replies

9. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

10. UNIX for Advanced & Expert Users

Duplicate lines in the file

Hi, I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me? Regards Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies

Featured Tech Videos