Sponsored Content
Top Forums Shell Programming and Scripting Filter uniq field values (non-substring) Post 302900749 by alister on Thursday 8th of May 2014 12:59:41 PM
Old 05-08-2014
Quote:
Originally Posted by yifangt
I get it now, actually I have hundred thousand lines.
That's the type of information that should always be mentioned in the initial post. Please keep that in mind going forward.

Quote:
Originally Posted by yifangt
I just thought another scenario: Is possible to do with two columns? i.e. any substring of the same column (but need both col2 and col4 at the same time), should be skipped.
Code:
infile:
1 abcd    idx01    ijklm
2 abc    idx03    klm
3 abcd    idx05    jkl
4 cdef    idx06    ijklm
5 efgh    idx07    abcd
6 efg    idx09    abc
7 efx    idx11    abcd
8 fgh    idx12    bcd

Code:
output:
1 abcd    idx01    ijklm
4 cdef    idx06    ijklm
5 efgh    idx07    abcd
7 efx    idx11    abcd

I tried using two arrays to loop:
Code:
gawk '{for(i in a) {if (index(i,$2)) next; for(j in b) {if (index(j,$4)) next; if (index($2, i) && index($4, j)) delete a[i]}; a[$2]=$0; b[$4]=$4}} END {for (i in a) print a[i]}' a[$2]=$0} END {for (i in a) print a[i]}'  infile

But there was no output. The second loop seems of problem, any suggestions please? Thanks a lot!
I haven't given it much thought, but upon cursory examination, your logic is definitely very flawed. If i is in a, you jump to the next line. That's wrong. If I understood the task, before you can skip a line, both i must be in a and j must be in b.

Once included, if later lines prove that an earlier line's $2 and $4 are substrings, then that earlier line must be excluded. Note that an earlier line's $2 and $4 may be disqualified by different subsequent lines, so you must track that as well.

Was there some copy-paste malfunction in your post? There are two END sections, nearly identical, which doesn't make sense (multiple END pattern-action pairs are allowed, but in this case I don't see the point of them).

One simple, if not optimal, way to solve the problem is to handle each column individually and then join the results.

Regards,
Alister
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Uniq using only the first field

Hi all, I have a file that contains a list of codes (shown below). I want to 'uniq' the file using only the first field. Anyone know an easy way of doing it? Cheers, Dave ##### Input File ##### 1xr1 1xws 1yxt 1yxu 1yxv 1yxx 2o3p 2o63 2o64 2o65 1xr1 1xws 1yxt 1yxv 1yxx 2o3p 2o63 2o64... (8 Replies)
Discussion started by: Digby
8 Replies

2. UNIX for Dummies Questions & Answers

How to uniq third field in a file

Hi ; I have a question regarding the uniq command in unix How do I uniq 3rd field in a file ? original file : zoom coord 39 18652 39 18652 zoom coord 39 18653 39 18653 zoom coord 39 18818 39 18818 zoom coord 39 18840 39 18840 zoom coord 41 15096 41 15096 zoom... (1 Reply)
Discussion started by: babycakes
1 Replies

3. Shell Programming and Scripting

How to use uniq on a certain field?

How can I use uniq on a certain field or what else could I use? If I want to use uniq on the second field and the output would remove one of the lines with a 5. bob 5 hand jane 3 leg jon 4 head chris 5 lungs (1 Reply)
Discussion started by: Bandit390
1 Replies

4. Shell Programming and Scripting

filter the uniq record problem

Anyone can help for filter the uniq record for below example? Thank you very much Input file 20090503011111|test|abc 20090503011112|tet1|abc|def 20090503011112|test1|bcd|def 20090503011131|abc|abc 20090503011131|bbc|bcd 20090503011152|bcd|abc 20090503011151|abc|abc... (8 Replies)
Discussion started by: bleach8578
8 Replies

5. Shell Programming and Scripting

Uniq based on first field

Hi New to unix. I want to display only the unrepeated lines from a file using first field. Ex: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236 uname5 status5 I used sort filename | uniq -u output: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236... (10 Replies)
Discussion started by: venummca
10 Replies

6. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

7. Shell Programming and Scripting

Printing uniq first field with the the highest second field

Hi All, I am searching for a script which will produce an output file with the uniq first field with the second field having highest value among all the duplicates.. The output file will produce only the uniqs which are duplicate 3 times.. Input file X 9 B 5 A 1 Z 9 T 4 C 9 A 4... (13 Replies)
Discussion started by: ailnilanjan
13 Replies

8. Shell Programming and Scripting

Grok filter to extract substring from path and add to host field in logstash

Hii, I am reading data from files by defining path as *.log etc, Files names are like app1a_test2_heep.log , cdc2a_test3_heep.log etc How to configure logstash so that the part of string that is string before underscore (app1a, cdc2a..) should be grepped and added to host field and... (7 Replies)
Discussion started by: Ravi Kishore
7 Replies

9. Shell Programming and Scripting

HELP - uniq values per column

Hi All, I am trying to output uniq values per column. see file below. can you please assist? Thank you in advance. cat names joe allen ibm joe smith ibm joe allen google joe smith google rachel allen google desired output is: joe allen google rachel smith ibm (5 Replies)
Discussion started by: Apollo
5 Replies

10. Shell Programming and Scripting

awk to update field using matching value in file1 and substring in field in file2

In the awk below I am trying to set/update the value of $14 in file2 in bold, using the matching NM_ in $12 or $9 in file2 with the NM_ in $2 of file1. The lengths of $9 and $12 can be variable but what is consistent is the start pattern will always be NM_ and the end pattern is always ;... (2 Replies)
Discussion started by: cmccabe
2 Replies
string(n)						       Tcl Built-In Commands							 string(n)

__________________________________________________________________________________________________________________________________________________

NAME
string - Manipulate strings SYNOPSIS
string option arg ?arg ...? _________________________________________________________________ DESCRIPTION
Performs one of several string operations, depending on option. The legal options (which may be abbreviated) are: string bytelength string Returns a decimal string giving the number of bytes used to represent string in memory. Because UTF-8 uses one to three bytes to represent Unicode characters, the byte length will not be the same as the character length in general. The cases where a script cares about the byte length are rare. In almost all cases, you should use the string length operation (including determining the length of a Tcl ByteArray object). Refer to the Tcl_NumUtfChars manual entry for more details on the UTF-8 representation. string compare ?-nocase? ?-length int? string1 string2 Perform a character-by-character comparison of strings string1 and string2. Returns -1, 0, or 1, depending on whether string1 is lexicographically less than, equal to, or greater than string2. If -length is specified, then only the first length characters are used in the comparison. If -length is negative, it is ignored. If -nocase is specified, then the strings are compared in a case- insensitive manner. string equal ?-nocase? ?-length int? string1 string2 Perform a character-by-character comparison of strings string1 and string2. Returns 1 if string1 and string2 are identical, or 0 when not. If -length is specified, then only the first length characters are used in the comparison. If -length is negative, it is ignored. If -nocase is specified, then the strings are compared in a case-insensitive manner. string first needleString haystackString ?startIndex? Search haystackString for a sequence of characters that exactly match the characters in needleString. If found, return the index of the first character in the first such match within haystackString. If not found, return -1. If startIndex is specified (in any of the forms accepted by the index method), then the search is constrained to start with the character in haystackString specified by the index. For example, string first a 0a23456789abcdef 5 will return 10, but string first a 0123456789abcdef 11 will return -1. string index string charIndex Returns the charIndex'th character of the string argument. A charIndex of 0 corresponds to the first character of the string. charIndex may be specified as follows: | integer | For any index value that passes string is integer -strict, the char specified at this integral index (e.g. 2 would refer | to the "c" in "abcd"). | end | The last char of the string (e.g. end would refer to the "d" in "abcd"). | end-N | The last char of the string minus the specified integer offset N (e.g. end-1 would refer to the "c" in "abcd"). | end+N | The last char of the string plus the specified integer offset N (e.g. end+-1 would refer to the "c" in "abcd"). | M+N | The char specified at the integral index that is the sum of integer values M and N (e.g. 1+1 would refer to the "c" in | "abcd"). | M-N | The char specified at the integral index that is the difference of integer values M and N (e.g. 2-1 would refer to the "b" | in "abcd"). | In the specifications above, the integer value M contains no trailing whitespace and the integer value N contains no leading white- | space. | If charIndex is less than 0 or greater than or equal to the length of the string then this command returns an empty string. | string is class ?-strict? ?-failindex varname? string Returns 1 if string is a valid member of the specified character class, otherwise returns 0. If -strict is specified, then an empty string returns 0, otherwise an empty string will return 1 on any class. If -failindex is specified, then if the function returns 0, the index in the string where the class was no longer valid will be stored in the variable named varname. The varname will not be set if the function returns 1. The following character classes are recognized (the class name can be abbreviated): alnum Any Unicode alphabet or digit character. alpha Any Unicode alphabet character. ascii Any character with a value less than u0080 (those that are in the 7-bit ascii range). boolean Any of the forms allowed to Tcl_GetBoolean. control Any Unicode control character. digit Any Unicode digit character. Note that this includes characters outside of the [0-9] range. double Any of the valid forms for a double in Tcl, with optional surrounding whitespace. In case of under/overflow in the value, 0 is returned and the varname will contain -1. false Any of the forms allowed to Tcl_GetBoolean where the value is false. graph Any Unicode printing character, except space. integer Any of the valid string formats for a 32-bit integer value in Tcl, with optional surrounding whitespace. In case of under/overflow in the value, 0 is returned and the varname will contain -1. list Any proper list structure, with optional surrounding whitespace. In case of improper list structure, 0 is returned and the varname will contain the index of the "element" where the list parsing fails, or -1 if this cannot be determined. lower Any Unicode lower case alphabet character. print Any Unicode printing character, including space. punct Any Unicode punctuation character. space Any Unicode space character. true Any of the forms allowed to Tcl_GetBoolean where the value is true. upper Any upper case alphabet character in the Unicode character set. | wideinteger | Any of the valid forms for a wide integer in Tcl, with optional surrounding whitespace. In case of under/overflow in | the value, 0 is returned and the varname will contain -1. wordchar Any Unicode word character. That is any alphanumeric character, and any Unicode connector punctuation characters (e.g. underscore). xdigit Any hexadecimal digit character ([0-9A-Fa-f]). In the case of boolean, true and false, if the function will return 0, then the varname will always be set to 0, due to the varied nature of a valid boolean value. string last needleString haystackString ?lastIndex? Search haystackString for a sequence of characters that exactly match the characters in needleString. If found, return the index of the first character in the last such match within haystackString. If there is no match, then return -1. If lastIndex is specified (in any of the forms accepted by the index method), then only the characters in haystackString at or before the specified lastIndex will be considered by the search. For example, string last a 0a23456789abcdef 15 will return 10, but string last a 0a23456789abcdef 9 will return 1. string length string Returns a decimal string giving the number of characters in string. Note that this is not necessarily the same as the number of bytes used to store the string. If the object is a ByteArray object (such as those returned from reading a binary encoded channel), then this will return the actual byte length of the object. string map ?-nocase? mapping string Replaces substrings in string based on the key-value pairs in mapping. mapping is a list of key value key value ... as in the form returned by array get. Each instance of a key in the string will be replaced with its corresponding value. If -nocase is speci- fied, then matching is done without regard to case differences. Both key and value may be multiple characters. Replacement is done in an ordered manner, so the key appearing first in the list will be checked first, and so on. string is only iterated over once, so earlier key replacements will have no affect for later key matches. For example, string map {abc 1 ab 2 a 3 1 0} 1abcaababcabababc will return the string 01321221. Note that if an earlier key is a prefix of a later one, it will completely mask the later one. So if the previous example is reordered like this, string map {1 0 ab 2 a 3 abc 1} 1abcaababcabababc it will return the string 02c322c222c. string match ?-nocase? pattern string See if pattern matches string; return 1 if it does, 0 if it does not. If -nocase is specified, then the pattern attempts to match against the string in a case insensitive manner. For the two strings to match, their contents must be identical except that the following special sequences may appear in pattern: * Matches any sequence of characters in string, including a null string. ? Matches any single character in string. [chars] Matches any character in the set given by chars. If a sequence of the form x-y appears in chars, then any character between x and y, inclusive, will match. When used with -nocase, the end points of the range are converted to lower case first. Whereas {[A-z]} matches "_" when matching case-sensitively (since "_" falls between the "Z" and "a"), with -nocase this is considered like {[A-Za-z]} (and probably what was meant in the first place). x Matches the single character x. This provides a way of avoiding the special interpretation of the characters *?[] in pattern. string range string first last Returns a range of consecutive characters from string, starting with the character whose index is first and ending with the charac- ter whose index is last. An index of 0 refers to the first character of the string. first and last may be specified as for the index method. If first is less than zero then it is treated as if it were zero, and if last is greater than or equal to the length of the string then it is treated as if it were end. If first is greater than last then an empty string is returned. string repeat string count Returns string repeated count number of times. string replace string first last ?newstring? Removes a range of consecutive characters from string, starting with the character whose index is first and ending with the charac- ter whose index is last. An index of 0 refers to the first character of the string. First and last may be specified as for the index method. If newstring is specified, then it is placed in the removed character range. If first is less than zero then it is treated as if it were zero, and if last is greater than or equal to the length of the string then it is treated as if it were end. If first is greater than last or the length of the initial string, or last is less than 0, then the initial string is returned untouched. | string reverse string | Returns a string that is the same length as string but with its characters in the reverse order. string tolower string ?first? ?last? Returns a value equal to string except that all upper (or title) case letters have been converted to lower case. If first is speci- fied, it refers to the first char index in the string to start modifying. If last is specified, it refers to the char index in the string to stop at (inclusive). first and last may be specified as for the index method. string totitle string ?first? ?last? Returns a value equal to string except that the first character in string is converted to its Unicode title case variant (or upper case if there is no title case variant) and the rest of the string is converted to lower case. If first is specified, it refers to the first char index in the string to start modifying. If last is specified, it refers to the char index in the string to stop at (inclusive). first and last may be specified as for the index method. string toupper string ?first? ?last? Returns a value equal to string except that all lower (or title) case letters have been converted to upper case. If first is speci- fied, it refers to the first char index in the string to start modifying. If last is specified, it refers to the char index in the string to stop at (inclusive). first and last may be specified as for the index method. string trim string ?chars? Returns a value equal to string except that any leading or trailing characters present in the string given by chars are removed. If chars is not specified then white space is removed (spaces, tabs, newlines, and carriage returns). string trimleft string ?chars? Returns a value equal to string except that any leading characters present in the string given by chars are removed. If chars is not specified then white space is removed (spaces, tabs, newlines, and carriage returns). string trimright string ?chars? Returns a value equal to string except that any trailing characters present in the string given by chars are removed. If chars is not specified then white space is removed (spaces, tabs, newlines, and carriage returns). string wordend string charIndex Returns the index of the character just after the last one in the word containing character charIndex of string. charIndex may be specified as for the index method. A word is considered to be any contiguous range of alphanumeric (Unicode letters or decimal dig- its) or underscore (Unicode connector punctuation) characters, or any single character other than these. string wordstart string charIndex Returns the index of the first character in the word containing character charIndex of string. charIndex may be specified as for the index method. A word is considered to be any contiguous range of alphanumeric (Unicode letters or decimal digits) or underscore (Unicode connector punctuation) characters, or any single character other than these. EXAMPLE
Test if the string in the variable string is a proper non-empty prefix of the string foobar. set length [string length $string] if {$length == 0} { set isPrefix 0 } else { set isPrefix [string equal -length $length $string "foobar"] } SEE ALSO
expr(n), list(n) KEYWORDS
case conversion, compare, index, match, pattern, string, word, equal, ctype, character, reverse Tcl 8.1 string(n)
All times are GMT -4. The time now is 03:43 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy