![]() |
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here. |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| concatenate all duplicate line in a file. | vaskarbasak | Shell Programming and Scripting | 30 | 08-29-2008 06:31 AM |
| reading text file line by line | MizzGail | Shell Programming and Scripting | 6 | 04-14-2008 07:58 AM |
| Eleminating Duplicate IPs from a text file | coolkid | Shell Programming and Scripting | 4 | 01-24-2008 07:22 PM |
| Duplicate records from oracle to text file. | shilendrajadon | UNIX for Advanced & Expert Users | 1 | 01-10-2008 11:21 AM |
| paging a text file line by line? | tomapam | UNIX for Dummies Questions & Answers | 2 | 10-07-2002 10:30 AM |
![]() |
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
|
|
||||
|
duplicate line in a text file
i would like to scan file in for duplicate lines, and print the duplicates to another file,
oh and it has to be case insensitive. example line1 line2 line2 line3 line4 line4 outputfile: line2 line4 any ideas |
|
||||
|
Code:
perl -ne '$l = lc(); print if $m{$l}++; ' file
lc returns the specified string in lowercase; in this case, with no parameter, it defaults to the current input line. $m{$l} is the count of number of times we have seen $l; if it's nonzero, it's a duplicate, so we print it. Last edited by era; 04-25-2008 at 02:50 PM.. Reason: Add explanation |
|
||||
|
Quote:
thanks it's worked for one file, but i forgot to add that if i need it to run on a directory and make sure to do in on a file only and not subdirectory what modifications would you do |
|
||||
|
Loop over files in a directory, and remove duplicates? Do you want to replace the files? Code:
for f in directory/*; do
test -d "$f" && continue # skip if it's a subdirectory
perl -ne '$l = lc(); print if $m{$l}++; ' "$f" >"$f.tmp"
mv "$f.tmp" "$f"
done
|
![]() |
| Bookmarks |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|