Sponsored Content
Full Discussion: Matching by key fields
Top Forums Shell Programming and Scripting Matching by key fields Post 302282706 by drl on Sunday 1st of February 2009 12:52:03 PM
Old 02-01-2009
Hi, CB.

You might benchmark these approaches to see what is best for your situation,

I would start with the smallest practical set to see what the overhead might be, then perhaps 1, 2, 5, and 10% of the data to see how it is progressing. Many complex solutions tend not to be linear. I think both will be somewhat memory intensive if you have 300K lines in your key.dat file.

Keep us posted ... cheers, drl

PS I used fgrep to avoid any interpretation of special characters in your patterns. The sample you provided used only "|" which is recognized by egrep, but I don't know how representative your sample was.

Last edited by drl; 02-01-2009 at 01:57 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Fill the Key fields : Please help us

Hi .... we are having the below file .Column 1, Column 2 ,column 3 are key fields... In the below ...for 2 nd , 3 rd row the repeated key column is missing .... i want the new file to be populated with all missing key columns. ... (11 Replies)
Discussion started by: charandevu
11 Replies

2. Shell Programming and Scripting

matching 2 exact fields

Dear experts, I have a file1 that looks like 60127930928 2091 60129382039 2092 60126382937 2091 60128937928 2061 60127329389 2062 60123748730 2061 60128730293 2061 and file 2 that looks like 60127930928 2091 60129382039 2092 60126382937 2093 60128937928 2061 60127329389... (2 Replies)
Discussion started by: aismann
2 Replies

3. Shell Programming and Scripting

Perl function to sort a file based on key fields

Hi, I am new to PERL.I want to sort all the lines in a file based on 1,2 and 4th filelds. Can U suggest me a command/function in perl for this operation.. (5 Replies)
Discussion started by: karthikd214
5 Replies

4. Shell Programming and Scripting

awk should output if one input file doesnt have matching key

nawk -F, 'FNR==NR{a= $3 ;next} $2 in a{print $1, 'Person',$2, a}' OFS=, filea fileb Input filea Input fileb output i am getting : (2 Replies)
Discussion started by: pinnacle
2 Replies

5. Linux

matching two fields

Hi I am having 2 fields and if f1=f2 i wanna print that line eg 1 2 1 3 1 9 2 2 3 5 9 9 In the abov eg. the highlighted lines shud be printed 2 2 9 9 Thanking u (3 Replies)
Discussion started by: binnybio
3 Replies

6. Shell Programming and Scripting

Compare Fields from two text files using key columns

Hi All, I have two files to compare. Each has 10 columns with first 4 columns being key index together. The rest of the columns have monetary values. Using Perl, I want to read one file into hash; check for the key value availability in file 2; then compare the values in the rest of 6... (2 Replies)
Discussion started by: Sangtha
2 Replies

7. Shell Programming and Scripting

Rsa public private key matching

Hi All, I have a requirement where i need to check if an rsa public key corresponds to a private key and hence return success or failure. Currently i am using the command diff <( ssh-keygen -y -e -f "$PRIVKEY" ) <( ssh-keygen -y -e -f "$PUBLICKEY" ) and its solving my purpose. This is in... (1 Reply)
Discussion started by: mritusmoi
1 Replies

8. UNIX for Dummies Questions & Answers

awk - Print lines if only matching key is found

I am looking to move matching lines (01 - 07) from File1 and 77 tab the matching string from File2, to File3.txt. I am almost done but - Currently, script is not printing lines to File3.txt in order. Thanks a lot. Any help is appreciated. Script I am using: awk 'FNR == NR && ! /^]*$/ {... (9 Replies)
Discussion started by: High-T
9 Replies

9. UNIX for Dummies Questions & Answers

File updation on matching key

I have input file like Input.dat with below content RRD 0Z91YUn000000Lk 9000100001 103020151117 STMT151117155527001 0000 2 000000 000004 RRD 0Z91YUn00000ysj 9000100001 103020151117 STMT151117155527001 0000 3 000000 000003 RRD 0Z91YUn00001vGh 9000100002... (12 Replies)
Discussion started by: PRAMOD 96
12 Replies

10. UNIX for Beginners Questions & Answers

Matching 2 files based on key

Hi all I have two files I need to match record from first file and second file on column 1,8 and and output only match records on file1 File1: 020059801803180116130926800002090000800231000245204003160000000002000461OUNCE000000350000100152500BM01007W0000 ... (5 Replies)
Discussion started by: arunkumar_mca
5 Replies
egrep(1)																  egrep(1)

NAME
egrep - search a file for a pattern using full regular expressions SYNOPSIS
/usr/bin/egrep [-bchilnsv] [-e pattern_list] [-f file] [strings] [file...] /usr/xpg4/bin/egrep [-bchilnsvx] [-e pattern_list] [-f file] [strings] [file...] The egrep (expression grep) utility searches files for a pattern of characters and prints all lines that contain that pattern. egrep uses full regular expressions (expressions that have string values that use the full set of alphanumeric and special characters) to match the patterns. It uses a fast deterministic algorithm that sometimes needs exponential space. If no files are specified, egrep assumes standard input. Normally, each line found is copied to the standard output. The file name is printed before each line found if there is more than one input file. /usr/bin/egrep The /usr/bin/egrep utility accepts full regular expressions as described on the regexp(5) manual page, except for ( and ), ( and ), { and }, < and >, and , and with the addition of: 1. A full regular expression followed by + that matches one or more occurrences of the full regular expression. 2. A full regular expression followed by ? that matches 0 or 1 occurrences of the full regular expression. 3. Full regular expressions separated by | or by a NEWLINE that match strings that are matched by any of the expressions. 4. A full regular expression that can be enclosed in parentheses ()for grouping. Be careful using the characters $, *, [, ^, |, (, ), and in full regular expression, because they are also meaningful to the shell. It is safest to enclose the entire full regular expression in single quotes '... '. The order of precedence of operators is [], then *?+, then concatenation, then | and NEWLINE. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility uses the regular expressions described in the EXTENDED REGULAR EXPRESSIONS section of the regex(5) manual page. The following options are supported for both /usr/bin/egrep and /usr/xpg4/bin/egrep: -b Precede each line by the block number on which it was found. This can be useful in locating block numbers by context (first block is 0). -c Print only a count of the lines that contain the pattern. -e pattern_list Search for a pattern_list (full regular expression that begins with a -). -f file Take the list of full regular expressions from file. -h Suppress printing of filenames when searching multiple files. -i Ignore upper/lower case distinction during comparisons. -l Print the names of files with matching lines once, separated by NEWLINEs. Does not repeat the names of files when the pat- tern is found more than once. -n Precede each line by its line number in the file (first line is 1). -s Work silently, that is, display nothing except error messages. This is useful for checking the error status. -v Print all lines except those that contain the pattern. /usr/xpg4/bin/egrep The following option is supported for /usr/xpg4/bin/egrep only: -x Consider only input lines that use all characters in the line to match an entire fixed string or regular expression to be matching lines. The following operands are supported: file A path name of a file to be searched for the patterns. If no file operands are specified, the standard input is used. /usr/bin/egrep pattern Specify a pattern to be used during the search for input. /usr/xpg4/bin/egrep pattern Specify one or more patterns to be used during the search for input. This operand is treated as if it were specified as -epattern_list. USAGE
See largefile(5) for the description of the behavior of egrep when encountering files greater than or equal to 2 Gbyte ( 2**31 bytes). See environ(5) for descriptions of the following environment variables that affect the execution of egrep: LC_COLLATE, LC_CTYPE, LC_MES- SAGES, and NLSPATH. The following exit values are returned: 0 If any matches are found. 1 If no matches are found. 2 For syntax errors or inaccessible files (even if matches were found). See attributes(5) for descriptions of the following attributes: /usr/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |CSI |Not Enabled | +-----------------------------+-----------------------------+ /usr/xpg4/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWxcu4 | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ fgrep(1), grep(1), sed(1), sh(1), attributes(5), environ(5), largefile(5), regex(5), regexp(5), XPG4(5) Ideally there should be only one grep command, but there is not a single algorithm that spans a wide enough range of space-time tradeoffs. Lines are limited only by the size of the available virtual memory. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility is identical to /usr/xpg4/bin/grep -E (see grep(1)). Portable applications should use /usr/xpg4/bin/grep -E. 23 May 2005 egrep(1)
All times are GMT -4. The time now is 01:02 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy