The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
extracting unique lines from text file extracting unique lines from text file
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
extracting lines from a file trichyselva UNIX for Dummies Questions & Answers 4 09-02-2008 06:18 AM
Extracting records with unique fields from a fixed width txt file sitney Shell Programming and Scripting 8 02-10-2008 03:18 AM
how to read all the unique words in a text file aditya.ece1985 Shell Programming and Scripting 5 11-30-2007 02:26 AM
Extracting the lines between 2 strings of a file babloo Shell Programming and Scripting 2 02-14-2007 11:27 AM
extracting lines from a file pradeepthanraj Shell Programming and Scripting 1 11-15-2005 09:41 AM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 08-22-2008
soliberus soliberus is offline
Registered User
  
 

Join Date: Oct 2004
Posts: 85
extracting unique lines from text file
I have a file with 14million lines and I would like to extract all the unique lines from the file into another text file.

For example:

Contents of file1

happy
sad
smile
happy
funny
sad

I want to run a command against file one that only returns the unique lines (ie 1 line for happy and 1 line for sad).

Could someone please point me in the right direction.

Thanks
  #2 (permalink)  
Old 08-22-2008
Franklin52 Franklin52 is offline Forum Staff  
Moderator
  
 

Join Date: Feb 2007
Posts: 4,212
You can try:

Code:
sort file | uniq > new_file
Regards
  #3 (permalink)  
Old 08-22-2008
danmero danmero is offline Forum Advisor  
丹
  
 

Join Date: Nov 2007
Location: 45.48-73.63
Posts: 1,369
If your sort support -u "uniq" option:
Code:
sort -u file > new_file
  #4 (permalink)  
Old 08-22-2008
soliberus soliberus is offline
Registered User
  
 

Join Date: Oct 2004
Posts: 85
Thank you...it works like a charm
Sponsored Links
Closed Thread

Bookmarks

« reading from 2 files through while loop | Gather File permission during scripting on unix as numbers. »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Linear Mode Linear Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 05:08 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language translation by Google.
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0