The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Dummies Questions & Answers
Google UNIX.COM
Home Forums Register Rules & FAQ Members List Arcade Search Today's Posts Mark Forums Read


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!


Other UNIX.COM Threads You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove Duplicate lines from File Nysif Steve UNIX for Dummies Questions & Answers 18 09-09-2007 05:57 AM
removing duplicate lines from a file ocelot UNIX for Dummies Questions & Answers 4 01-25-2007 07:02 AM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 02:28 AM
Remove Duplicate Lines in File Teh Tiack Ein Shell Programming and Scripting 5 01-12-2006 04:30 AM
Counting The Number Of Duplicate Lines In a File crunchtime UNIX for Dummies Questions & Answers 2 07-04-2003 10:24 AM

Reply
 
Submit Tools LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 04-23-2007
Registered User
 

Join Date: Mar 2007
Posts: 33
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiReddit! Stumble this Post!Spurl this Post!
How to redirect duplicate lines from a file????

Hi,

I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file.

Suppose I have a file called file_dup.txt which contains some line as

file_dup.txt

A100-R1
ACCOUNTING-CONTROL
ACTONA-ACTASTOR
ADMIN-AUTH-STATS
ACTONA-ACTASTOR
ADMIN-AUTH-STATS

If you see the above file contents the 3 and 4 numbered lines are duplicated. I want to redirect the lines which are duplicated in to another file say file_nodup.txt. So the file_nodup.txt will contain the line as follows

file_nodup.txt

ACTONA-ACTASTOR
ADMIN-AUTH-STATS

Is there any command in UNIX which will give me the above work done?
Or any AWK way to solve it.

Thanks in advance…

Regards…
zing_foru
Reply With Quote
Forum Sponsor
  #2 (permalink)  
Old 04-23-2007
Registered User
 

Join Date: Mar 2006
Location: Bangalore,India
Posts: 1,397
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiReddit! Stumble this Post!Spurl this Post!
Code:
awk ' ++arr[$0] > 1 ' file_dup.txt > file_nodup.txt
Reply With Quote
  #3 (permalink)  
Old 04-25-2007
Registered User
 

Join Date: Mar 2007
Posts: 19
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiReddit! Stumble this Post!Spurl this Post!
I think it can work by this way also:
sort file1 |uniq -d > file2
this will extract the duplicated lines and redirects them to file2

you can also specify the number of duplication of every line at the begining of the line:
sort file1 |uniq -c > file2
Reply With Quote
  #4 (permalink)  
Old 04-25-2007
Registered User
 

Join Date: Mar 2007
Posts: 33
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiReddit! Stumble this Post!Spurl this Post!
Hey thanks for all of you who suggested very good options to solve my problem.
Reply With Quote
Google UNIX.COM
Reply



Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -7. The time now is 05:46 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008 The CEP Blog All Rights Reserved -Ad Management by RedTyger

Search Engine Optimization by vBSEO 3.1.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102