I created a file1.txt with 500,000 keys and did some timing tests on my laptop (2.6GHz i3 M 350): awk takes about 1.08 secs to check a zip record against the 500K keys. I loaded the keys into a linked list in C and did a substring match an this took 0.842 secs per zip record (so with a custom C program you might expect a 15%-20% imporvement in speed).
The real issue here is the substring match of each key, if you could determine the key of a zip datafile record and do and database or B-tree lookup in would only take a few miliseconds per zip datafile record. For example if you could say the key for data record:
Was "56456456546" and indexed search could be done against your 500K keys and the matching key ticked off as found very quickly. Without this ability you are could really only expect to check in the order of 10 ziprecords per second.
Yesterday, I tried loading the keys into a c++ B-Tree and then pulling a key from your zip-string (11 chars from position 60) and doing a lookup. This managed to check 20,000 zip records in approx 0.7 sec. This highlights the power of indexed lookups, more work should probably be done in parsing your data records and determining the key from the record.
From the example zip record in my previous post we could lookup:
Depending on the region where data appears and what limitation you have on key characters (eg are spaces or letters allowed in the keys), up to 27 lookups would be required for each record, this should still be able to process close to 1000 records per second.
As you can see it's important to find out as much about your data as possible, Ideally the processing program should look at a record and identify it's type and extract the proper key exactly and report errors on badly formatted records. This avoids false positives, ensures you are looking up what is intended and can react to format changes within the data records.
As methyl pointed out earlier getting a Systems Analyst involved is your best bet as we can only advise on information you supply here and there may be even better ways to get the result you require.
Last edited by Chubler_XL; 01-26-2011 at 06:00 PM..
@vsachan
What Database Engine and version do you have?
Did the flat file data come from an extract from you main database(s)?
Have you now developed code to match your requirements?
Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values.
for date in {1..31}
do
for val in... (1 Reply)
Hi,
I have scenario like below and need to search for multiple patterns
Eg:
Test
Time Started= secs
Time Ended = secc
Green test
Test
Time Started= secs
Time Ended = secc
Green test
Output:
I need to display the text starting with Test and starting with Time... (2 Replies)
Hi All,
I am starting a service which will redirect its out put into 2 logs say A and B.
Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly.
Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Hi,
I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command.
Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns.
Xargs -I {} grep... (3 Replies)
I have a list of files all over a file system e.g.
/home/1/foo/bar.x
/www/sites/moose/foo.txtI'm looking for strings in these files and want to replace each occurrence with a replacement string, e.g.
if I find: '#@!^\&@ in any of the files I want to replace it with: 655#@11, etc.
There... (2 Replies)
Hi,
I tried to search multiple pattern using awk
trans=1234
reason=LN MISMATCH
rec=`awk '/$trans/ && /'"$reason"'/' file`
whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty
... (6 Replies)
I have two lists in a file that look like
a b
b a
e f
c d
f e
d c
I would like a final list
a b
c d
e f
I've tried multiple grep and awk but can't get it to work (8 Replies)
Good day, great gurus,
I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Hi
I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns.
Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
I have one file:
123*100*abcd*10
123*101*abcd*-29*def
123*100*abcd*-10
123*102*abcd*-105*asd
I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern.
*100* and *- (ie *minus)
*102* and *-
The output that is... (6 Replies)