Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Delete all occurences of a pattern except first

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete all occurences of a pattern except first
# 1  
Old 03-06-2012
Delete all occurences of a pattern except first
Hi,

Have a flat file like below:
1
2
4
1
2
1
3

Need to delete all duplicate instances.
Output:
1
2
4
3

Can't use "uniq" as it works on a sorted file.
Can't "sort" as need to maintain the original order.

Thanks in advance.

---------- Post updated at 03:51 PM ---------- Previous update was at 03:47 PM ----------

Gotcha
awk ' !x[$0]++'
# 2  
Old 03-06-2012
Actually awk '!x[$1]++' may be more reliable depending on your requirements as it avoids whitespace issues...
# 3  
Old 03-06-2012
perl -i.bak -ne 'print if ! $seen{$_}++;' test.dat has the advantage of doing the deletion in place (and making a backup in case of error)
# 4  
Old 03-06-2012
Can somebody explain how does the awk snippet work Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to delete all lines before a particular pattern when the pattern is defined in a variable?

I have a file Line 1 a Line 22 Line 33 Line 1 b Line 22 Line 1 c Line 4 Line 5 I want to delete all lines before last occurrence of a line which contains something which is defined in a variable. Say a variable var contains 'Line 1', then I need the following in the output. ... (21 Replies)
Discussion started by: Soham
21 Replies

2. Shell Programming and Scripting

If first pattern is found, look for second pattern. If second pattern not found, delete line

I had a spot of trouble coming up with a title, hopefully you'll understand once you read my problem... :) I have the output of an ldapsearch that looks like this: dn: cn=sam,ou=company,o=com uidNumber: 7174 gidNumber: 49563 homeDirectory: /home/sam loginshell: /bin/bash uid: sam... (2 Replies)
Discussion started by: samgoober
2 Replies

3. Shell Programming and Scripting

Count of matched pattern occurences by minute and date in a log file

Anyone knows how to use AWK to achieve the following Sun Feb 12 00:41:01-00:41:59 Success:2 Fail:2 Sun Feb 12 00:42:01-00:42:59 Success:1 Fail:2 Sun Feb 12 01:20:01-01:20:59 Success:1 Fail:2 Mon Feb 13 22:41:01-22:41:59 Success:1 Fail:1 log file: Success Success Fail Fail ... (9 Replies)
Discussion started by: timmywong
9 Replies

4. UNIX for Dummies Questions & Answers

Count pattern occurences

hi, I have a text..and i need to find a pattern in the text and count to the no of times the pattern occured. i have used grep command ..but the problem is , it shows the occurrences of the pattern but doesn't count no of times the pattern occuries. (5 Replies)
Discussion started by: nvnni
5 Replies

5. Shell Programming and Scripting

sed pattern to delete lines containing a pattern, except the first occurance

Hello sed gurus. I am using ksh on Sun and have a file created by concatenating several other files. All files contain header rows. I just need to keep the first occurrence and remove all other header rows. header for file 1111 2222 3333 header for file 1111 2222 3333 header for file... (8 Replies)
Discussion started by: gary_w
8 Replies

6. Shell Programming and Scripting

How can i delete the content between all the occurences of two strings using sed or awk command

Hi. I have to delete the content between all the occurrences of the xml tags in a single file. For example: * The tags <script>.....................</script> occurs more than once in the same file. * It follows tagging rules meaning a start tag will be followed by an end tag. Will not have... (9 Replies)
Discussion started by: satheeshkumar
9 Replies

7. Shell Programming and Scripting

Count of matched pattern occurences by time in a log file

We have a log file, the format is similar to this: 08/04/2011 05:03:08 Connection Success 08/04/2011 05:13:18 Connection Success 08/04/2011 05:23:28 Connection Fail 08/04/2011 05:33:38 Connection Success 08/04/2011 06:14:18 Connection Success 08/04/2011 06:24:28 Connection Fail 08/04/2011... (6 Replies)
Discussion started by: clu
6 Replies

8. Shell Programming and Scripting

delete last character in all occurences of string

Hello all, I have a file containing the following p1 q1 p2 q2 p1 p2 p3 pr1 pr2 pr1 pr2 pa1 pa2 I want to remove the last character from all strings that start with 'p' and end with '1'. In general, I do not know what is between the first part of the string and the last part of the string.... (4 Replies)
Discussion started by: bigfoot
4 Replies

9. Shell Programming and Scripting

How to count the number of occurences of this pattern?

Hi all, I have a pattern like this in a file: 123 4 56 789 234 5 67 789 121 3 56 789 222 4 65 789 321 6 90 100 478 8 40 789 243 7 80 789 How can I count the number of occurences of '789' (4th column) in this set...? Thanks for all your help! K (7 Replies)
Discussion started by: kripssmart
7 Replies

10. Shell Programming and Scripting

comment/delete a particular pattern starting from second line of the matching pattern

Hi, I have file 1.txt with following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433 ** ** ** In file 2.txt I have the following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433... (4 Replies)
Discussion started by: imas
4 Replies
Login or Register to Ask a Question