Sponsored Content
Top Forums Shell Programming and Scripting Search for duplicates and delete but remain the first one based on a specific pattern Post 302837861 by redse171 on Saturday 27th of July 2013 08:22:57 AM
Old 07-27-2013
Search for duplicates and delete but remain the first one based on a specific pattern

Hi all,

I have been trying to delete duplicates based on a certain pattern but failed to make it works. There are more than 1 pattern which are duplicated but i just want to remove 1 pattern only and remain the rest. I cannot use awk '!x[$0]++' inputfile.txt or sed '/pattern/d' or use uniq and sort command as it will deleted all the duplicated patterns in the file. A sample as follows:

inputfile.txt
Code:
;;  
;;
ID    701
NAME    701
FUNC    Null
FUNC    Null
FUNC    Null
CC    27749
PRO    A
NO    NO:3676
NO    NO:3677
NO    NO:3723
NO    NO:3964
COMMENT    Nothing is impossible
@@
ID    702
NAME    702
FUNC    Null
FUNC    Null
FUNC    Null
FUNC    Null
PRO    A
NO    NO:3676
NO    NO:3677
COMMENT    Need to change
@@
ID    706
NAME    706
FUNC    Null
PRO    A
NO    NO:6301
NO    NO:6310
NO    NO:6450
NO    NO:6647
NO    NO:6812
@@

I want to remove the duplicates for pattern "FUNC" only, where the output should look like this:

output.txt
Code:
;;  
;;
ID    701
NAME    701
FUNC    Null
CC    27749
PRO    A
NO    NO:3676
NO    NO:3677
NO    NO:3723
NO    NO:3964
COMMENT    Nothing is impossible
@@
ID    702
NAME    702
FUNC    Null
PRO    A
NO    NO:3676
NO    NO:3677
COMMENT    Need to change
@@
ID    706
NAME    706
FUNC    Null
PRO    A
NO    NO:6301
NO    NO:6310
NO    NO:6450
NO    NO:6647
NO    NO:6812
@@

I have thousands of data like this and i need to delete a different pattern at one time. I tried to do it by specifying the column no too but it affects other duplicated values which i dont want it to be affected. Appreciate your help on this. Thanks
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete and remain 2 value

Hello Friend, I have the followint command to delete 4th field and move forward. Can I delete all filed and just remain the first 2? sed -e "/^*<Number/s/\(\) \(\)/\1\2/g" -e "/^*<Number/s/\(\)./\1/" -e "/^*<Number/s/\(\)/\1 /g" -e "/^*<Number/s/0</</" file input <Number>00000000<Number>... (5 Replies)
Discussion started by: happyv
5 Replies

2. UNIX for Dummies Questions & Answers

Search for very specific pattern with less

Hi, I want to search a certain pattern with less command in a files. For examples, I have a files with this entry: POLAR xx POLARX xc POLARXI x1 POLARZZZY vb POLARLLLLLLL ee... (1 Reply)
Discussion started by: anjas
1 Replies

3. Shell Programming and Scripting

How can i delete the duplicates based on one column of a line

I have my data something like this (08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb (08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa (08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts (08/03/2009 22:57:42.425)(:) Ravi... (11 Replies)
Discussion started by: rdhanek
11 Replies

4. HP-UX

How to delete specific pattern in a file with SED?

I have one file which is having content as following... 0513468211,,,,20091208,084005,5,,2,3699310, 0206554475,,,,20090327,123634,85,,2,15615533 0206554475,,,,20090327,134431,554,,2,7246177 0103000300,,,,20090523,115501,89,,2,3869929 0736454328,,,,20091208,084005,75,,2,3699546... (7 Replies)
Discussion started by: ganesh.mandlik
7 Replies

5. Shell Programming and Scripting

Trimming sequences based on specific pattern

My files look like this And I need to cut the sequences at the last "A" found in the following 'pattern' -highlighted for easier identification, the pattern is the actual file is not highlighted. The expected result should look like this Thus, all the sequences would end with AGCCCTA... (2 Replies)
Discussion started by: Xterra
2 Replies

6. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

7. Shell Programming and Scripting

Print a pattern between the xml tags based on a search pattern

Hi all, I am trying to extract the values ( text between the xml tags) based on the Order Number. here is the sample input <?xml version="1.0" encoding="UTF-8"?> <NJCustomer> <Header> <MessageIdentifier>Y504173382</MessageIdentifier> ... (13 Replies)
Discussion started by: oky
13 Replies

8. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ... (13 Replies)
Discussion started by: script_op2a
13 Replies

9. Shell Programming and Scripting

Delete files based on specific MMDDYYYY pattern in filename

Hi Unix gurus, I am trying to remove the filenames based on MMDDYYYY in the physical name as such so that the directory always has the recent 3 files based on MMDDYYYY. "HHMM" is just dummy in this case. You wont have two files with different HHMM on the same day. For example in a... (4 Replies)
Discussion started by: shankar1dada
4 Replies

10. Shell Programming and Scripting

Finding duplicates in a file excluding specific pattern

I have unix file like below >newuser newuser <hello hello newone I want to find the unique values in the file(excluding <,>),so that the out put should be >newuser <hello newone can any body tell me what is command to get this new file. (7 Replies)
Discussion started by: shiva2985
7 Replies
switch(n)						       Tcl Built-In Commands							 switch(n)

__________________________________________________________________________________________________________________________________________________

NAME
switch - Evaluate one of several scripts, depending on a given value SYNOPSIS
switch ?options? string pattern body ?pattern body ...? switch ?options? string {pattern body ?pattern body ...?} _________________________________________________________________ DESCRIPTION
The switch command matches its string argument against each of the pattern arguments in order. As soon as it finds a pattern that matches string it evaluates the following body argument by passing it recursively to the Tcl interpreter and returns the result of that evaluation. If the last pattern argument is default then it matches anything. If no pattern argument matches string and no default is given, then the switch command returns an empty string. If the initial arguments to switch start with - then they are treated as options. The following options are currently supported: -exact Use exact matching when comparing string to a pattern. This is the default. -glob When matching string to the patterns, use glob-style matching (i.e. the same as implemented by the string match command). -regexp When matching string to the patterns, use regular expression matching (as described in the re_syntax reference page). -- Marks the end of options. The argument following this one will be treated as string even if it starts with a -. Two syntaxes are provided for the pattern and body arguments. The first uses a separate argument for each of the patterns and commands; this form is convenient if substitutions are desired on some of the patterns or commands. The second form places all of the patterns and commands together into a single argument; the argument must have proper list structure, with the elements of the list being the patterns and commands. The second form makes it easy to construct multi-line switch commands, since the braces around the whole list make it unnec- essary to include a backslash at the end of each line. Since the pattern arguments are in braces in the second form, no command or vari- able substitutions are performed on them; this makes the behavior of the second form different than the first form in some cases. If a body is specified as ``-'' it means that the body for the next pattern should also be used as the body for this pattern (if the next pattern also has a body of ``-'' then the body after that is used, and so on). This feature makes it possible to share a single body among several patterns. Beware of how you place comments in switch commands. Comments should only be placed inside the execution body of one of the patterns, and not intermingled with the patterns. Below are some examples of switch commands: switch abc a - b {format 1} abc {format 2} default {format 3} will return 2, switch -regexp aaab { ^a.*b$ - b {format 1} a* {format 2} default {format 3} } will return 1, and switch xyz { a - b { # Correct Comment Placement format 1 } a* {format 2} default {format 3} } will return 3. SEE ALSO
for(n), if(n), regexp(n) KEYWORDS
switch, match, regular expression Tcl 7.0 switch(n)
All times are GMT -4. The time now is 11:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy