Sponsored Content
Top Forums Shell Programming and Scripting Kindly check:remove duplicates with similar data in front of it Post 302678525 by rangarasan on Saturday 28th of July 2012 07:26:08 AM
Old 07-28-2012
awk

Hi,
The input file has some pattern match related characters like []. I have not tested the below code. Make a try with this.
Code:
awk '{$0=gensub(/([\]\[\(\)\{\}])/,"\\\1","g",$0);t=$0;r=$1"";sub(r,"",t);if(a[$1]!~t){a[$1]=a[$1]""t;}else{if(!a[$1]){a[$1]=t;}}}END{for(i in a){print i,a[i];}}' file1

you have to escape the special characters before going to use those in regex. You can also use quotemeta function in perl and then pass those output lines to awk.
Cheers,
Ranga:-)

Last edited by rangarasan; 07-28-2012 at 08:40 AM.. Reason: add perl func name
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove space in front or end of each field

Hi, I have a txt file called a.txt which contain over 10,000 records and I would like to remove space before comma or after comma....like below: The input (for example two record 00001,00002): 00001,client,card limited ,02292,N ,162:41 , 192, ... (6 Replies)
Discussion started by: happyv
6 Replies

2. Shell Programming and Scripting

sh, ksh: command to remove front spaces from a string?

dear pro-coders, is there any command out there that takes out the front spaces from a string? sample strings: 4 members 5 members 3 members but it has to be like so: 4 members 5 members 3 members (3 Replies)
Discussion started by: pseudocoder
3 Replies

3. Shell Programming and Scripting

Request to check:Remove duplicates

Hi all I have a file with following kind of data I want to remove duplicates according to first column so that output contains Kindly let me scripting regading this. (4 Replies)
Discussion started by: manigrover
4 Replies

4. Shell Programming and Scripting

Request to check remove duplicates but write before it

Hi alll I have a file with following kind input I want in output duplicates should not be there but there should be numbering mentioned before that like (4 Replies)
Discussion started by: manigrover
4 Replies

5. Shell Programming and Scripting

Request to check:remove duplicates and write sytematically

Hi all I have a file with following input It contains 5 columns gene name drug drug ID disease approved Now the same gene is repeated many times with different data in column2,3 ,4,5 I want to arrange dat in such a way that there shuld be one entry in the column(no... (2 Replies)
Discussion started by: manigrover
2 Replies

6. Shell Programming and Scripting

Request to check:remove duplicates only in first column

Hi all, I have an input file like this Now I have to remove duplicates only in first column and nothing has to be changed in second and third column. so that output would be Please let me know scripting regarding this (20 Replies)
Discussion started by: manigrover
20 Replies

7. Shell Programming and Scripting

Kindly check it: Camparison of files only column1 of 2 files

Hi all, I have 2 files in which i have to find commom entries in column 1 an dif soemthing is common write other data of both files in front of it mentioned. Gene symbol and disease name column 1 column2 ARFGEF2 CAD DDEF2 CAD PSCD3 CAD PSCD4 CAD CAMK1... (15 Replies)
Discussion started by: manigrover
15 Replies

8. Shell Programming and Scripting

common entries of first column in 2 or 3 files:kindly check

Hi all, I have 3 files with such data first files second file third file I have to find common entries of first column in two ways 1) between 2 files (2 Replies)
Discussion started by: manigrover
2 Replies

9. Shell Programming and Scripting

Sort data by date first and then remove duplicates

Hi , I have below data inside a file named ref.psv . I want to create a shell script which will do the below 2 points : (1) sort the file content first based on the latest date which is the last column in the file (actual file its the 175th column) (2)after sorting the file based on latest date... (3 Replies)
Discussion started by: samrat dutta
3 Replies
regex(1F)                                                          FMLI Commands                                                         regex(1F)

NAME
regex - match patterns against a string SYNOPSIS
regex [-e] [ -v "string"] [ pattern template] ... pattern [template] DESCRIPTION
The regex command takes a string from the standard input, and a list of pattern / template pairs, and runs regex() to compare the string against each pattern until there is a match. When a match occurs, regex writes the corresponding template to the standard output and returns TRUE. The last (or only) pattern does not need a template. If that is the pattern that matches the string, the function simply returns TRUE. If no match is found, regex returns FALSE. The argument pattern is a regular expression of the form described in regex(). In most cases, pattern should be enclosed in single quotes to turn off special meanings of characters. Note that only the final pattern in the list may lack a template. The argument template may contain the strings $m0 through $m9, which will be expanded to the part of pattern enclosed in ( ... )$0 through ( ... )$9 constructs (see examples below). Note that if you use this feature, you must be sure to enclose template in single quotes so that FMLI does not expand $m0 through $m9 at parse time. This feature gives regex much of the power of cut(1), paste(1), and grep(1), and some of the capabilities of sed(1). If there is no template, the default is $m0$m1$m2$m3$m4$m5$m6$m7$m8$m9. OPTIONS
The following options are supported: -e Evaluates the corresponding template and writes the result to the standard output. -v "string" Uses string instead of the standard input to match against patterns. EXAMPLES
Example 1: Cutting letters out of a string To cut the 4th through 8th letters out of a string (this example will output strin and return TRUE): `regex -v "my string is nice" '^.{3}(.{5})$0' '$m0'` Example 2: Validating input in a form In a form, to validate input to field 5 as an integer: valid=`regex -v "$F5" '^[0-9]+$'` Example 3: Translating an environment variable in a form In a form, to translate an environment variable which contains one of the numbers 1, 2, 3, 4, 5 to the letters a, b, c, d, e: value=`regex -v "$VAR1" 1 a 2 b 3 c 4 d 5 e '.*' 'Error'` Note the use of the pattern '.*' to mean "anything else". Example 4: Using backquoted expressions In the example below, all three lines constitute a single backquoted expression. This expression, by itself, could be put in a menu defini- tion file. Since backquoted expressions are expanded as they are parsed, and output from a backquoted expression (the cat command, in this example) becomes part of the definition file being parsed, this expression would read /etc/passwd and make a dynamic menu of all the login ids on the system. `cat /etc/passwd | regex '^([^:]*)$0.*$' ' name=$m0 action=`message "$m0 is a user"`'` DIAGNOSTICS
If none of the patterns match, regex returns FALSE, otherwise TRUE. NOTES
Patterns and templates must often be enclosed in single quotes to turn off the special meanings of characters. Especially if you use the $m0 through $m9 variables in the template, since FMLI will expand the variables (usually to "") before regex even sees them. Single characters in character classes (inside []) must be listed before character ranges, otherwise they will not be recognized. For exam- ple, [a-zA-Z_/] will not find underscores (_) or slashes (/), but [_/a-zA-Z] will. The regular expressions accepted by regcmp differ slightly from other utilities (that is, sed, grep, awk, ed, and so forth). regex with the -e option forces subsequent commands to be ignored. In other words, if a backquoted statement appears as follows: `regex -e ...; command1; command2` command1 and command2 would never be executed. However, dividing the expression into two: `regex -e ...``command1; command2` would yield the desired result. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
awk(1), cut(1), grep(1), paste(1), sed(1), regcmp(3C), attributes(5) SunOS 5.10 12 Jul 1999 regex(1F)
All times are GMT -4. The time now is 11:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy