Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Delete duplicate lines... with a twist!
Top Forums Shell Programming and Scripting Delete duplicate lines... with a twist! Post 302575999 by durden_tyler on Wednesday 23rd of November 2011 11:02:14 AM
Old 11-23-2011
Code:
$
$ cat f42
"Heh, heh. Those darn ninjas. They're _____."*wacky
The "canebrake", "timber" & "pygmy" are types of what?*rattlesnakes
Science : The second space shuttle was named ------*challenger
the quick brown 123 fox jumps over the lazy ?@! dog
the 456 quick brown fox jumps over the ~*%# lazy dog
123 the quick brown @%#$!^ fox jumps over the lazy ~()& dog
$
$
$
$ perl -lne '$h=$_; s/[^\w]|_//g; tr/A-Z/a-z/; s/(.)(?=.*?\1)//g;
             $_=join "",sort split "";
             print $h if not defined $x{$_}; $x{$_}++
            ' f42
"Heh, heh. Those darn ninjas. They're _____."*wacky
The "canebrake", "timber" & "pygmy" are types of what?*rattlesnakes
Science : The second space shuttle was named ------*challenger
the quick brown 123 fox jumps over the lazy ?@! dog
the 456 quick brown fox jumps over the ~*%# lazy dog
$
$
$

tyler_durden
This User Gave Thanks to durden_tyler For This Post:
shadowww
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete semi-duplicate lines from file?

Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'" However, on a couple of systems, they have some mount points looking like this: /stage /stand /usr /MFPIS... (2 Replies)
Discussion started by: paqman
2 Replies

2. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

3. UNIX for Dummies Questions & Answers

How to delete or remove duplicate lines in a file

Hi please help me how to remove duplicate lines in any file. I have a file having huge number of lines. i want to remove selected lines in it. And also if there exists duplicate lines, I want to delete the rest & just keep one of them. Please help me with any unix commands or even fortran... (7 Replies)
Discussion started by: reva
7 Replies

4. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

5. UNIX for Dummies Questions & Answers

How to delete partial duplicate lines unix

hi :) I need to delete partial duplicate lines I have this in a file sihp8027,/opt/cf20,1980182 sihp8027,/opt/oracle/10gRelIIcd,155200016 sihp8027,/opt/oracle/10gRelIIcd,155200176 sihp8027,/var/opt/ERP,10376312 and need to leave it like this: sihp8027,/opt/cf20,1980182... (2 Replies)
Discussion started by: C|KiLLeR|S
2 Replies

6. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

7. Shell Programming and Scripting

Delete lines in file containing duplicate strings, keeping longer strings

The question is not as simple as the title... I have a file, it looks like this <string name="string1">RZ-LED</string> <string name="string2">2.0</string> <string name="string2">Version 2.0</string> <string name="string3">BP</string> I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies

8. Shell Programming and Scripting

Delete duplicate rows

Hi, This is a followup to my earlier post him mno klm 20 76 . + . klm_mango unix_00000001; alp fdc klm 123 456 . + . klm_mango unix_0000103; her tkr klm 415 439 . + . klm_mango unix_00001043; abc tvr klm 20 76 . + . klm_mango unix_00000001; abc def klm 83 84 . + . klm_mango... (5 Replies)
Discussion started by: jacobs.smith
5 Replies

9. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

10. UNIX for Beginners Questions & Answers

Delete duplicate like pattern lines

Hi I need to delete duplicate like pattern lines from a text file containing 2 duplicates only (one being subset of the other) using sed or awk preferably. Input: FM:Chicago:Development FM:Chicago:Development:Score SR:Cary:Testing:Testcases PM:Newyork:Scripting PM:Newyork:Scripting:Audit... (6 Replies)
Discussion started by: tech_frk
6 Replies

LEARN ABOUT DEBIAN

ha

ha(1)								    HA archiver 							     ha(1)

NAME

       ha - a small file archiver utility

SYNOPSIS

       ha <command>[<option><option>...]  archive [ name ...  ]

DESCRIPTION

       After the program name comes a command name (single letter) and options (single letters, appended to the command name without a space).

COMMANDS

       a[sdqemr012]
	      Add files matching search pattern to archive.

       e[aqty]
	      Extract files matching search pattern from archive.

       x[aqty]
	      Extract files matching search pattern from archive using path information stored in archive.

       l[f]   List files currently in archive.

       d[q]   Delete files matching search pattern from archive.  If archive does not contain any files after deletion it is removed.

       f[sdqemr012]
	      Freshen files in archive. All files matching search pattern and newer than version already in archive are updated to archive.

       u[sdqemr012]
	      Update  files to archive. All files matching search pattern that are newer than version already in archive or are not yet in archive
	      are updated to archive.

       t[q]   Test files in archive.

OPTIONS

       0,1,2  Try compression method #. More than one can be specified.

	      Methods are:
		0-CPY  Simple copying of files.
		1-ASC Default method using sliding window dictionary followed by arithmetic
		      coder. Offers quite good compression on wide variety of file types.
		2-HSC Compression method based on finite context model and arithmetic
		      coder. Quite slow for binary data, but offers very good compression
		      especially for longer text files.

       y      Assume answer yes on all questions.

       t      Touch files when extracting. Sets file times to current machine time.

       r      Recurse through subdirectories while searching for files matching search pattern.

       f      Full listing of stored file information.

       m      Move files when archiving. Files are removed from disk after they are successfully written to archive.

       a      Set system specific file attributes when extracting files.

       s      Find special files when archiving.

       e      Do not store pathname information.

       q      Quiet operation.

       d      Make separate entries for directories. Saves information about empty directories, directory attributes etc. in archive.

AUTHOR

       This manual page was written by Mikhail Gusarov <dottedmag@dottedmag.net> for Debian GNU/Linux system (but may be used by others).

ha 0.999p							    3 Jul 2008								     ha(1)
All times are GMT -4. The time now is 09:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy