![]() |
|
|
|
|
|||||||
| Forums | Portal | Register | Forum Rules | FAQ | Contribute | Members List | Arcade | Search | Today's Posts | Mark Forums Read |
| Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts here. |
|
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Remove duplicate lines in log files | karthikn7974 | Shell Programming and Scripting | 3 | 1 Day Ago 04:17 PM |
| Remove all instances of duplicate records from the file | vukkusila | Shell Programming and Scripting | 3 | 12-12-2007 04:50 AM |
| Remove Duplicate lines from File | Nysif Steve | UNIX for Dummies Questions & Answers | 18 | 09-09-2007 05:57 AM |
| how to remove duplicate lines | fredao | Shell Programming and Scripting | 3 | 12-13-2006 09:51 AM |
| Duplicate lines in the file | guptan | UNIX for Advanced & Expert Users | 3 | 05-18-2006 02:28 AM |
|
|
Submit Tools | LinkBack | Thread Tools | Search this Thread | Display Modes |
|
#1
|
|||
|
|||
|
Remove Duplicate Lines in File
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.
FileA ------ 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 ........ and I want to simply it with no duplicate line as file below. FileA ------ 1253-6856 3101-4011 1827-1356 1822-1157 1000-1410 1822-1231 ........ How can I do that in KSH? |
| Forum Sponsor | ||
|
|
|
#2
|
|||
|
|||
|
Try the following command:
Code:
awk '!x[$0]++' file > file.new |
|
#3
|
|||
|
|||
|
Thanks Nir! This is awesome!
|
|
#4
|
|||
|
|||
|
uniq file > file.new
If the files are not already sorted then: sort file | uniq > file.new |
|
#5
|
|||
|
|||
|
sort -u filename > filename.new
|
|
#6
|
|||
|
|||
|
Thanks
Thanks ssk!
Your solution is so simple and so effective! Thanks, Sumit Garg |
|||
| Google The UNIX and Linux Forums |