Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Search multiple patterns in multiple files
Top Forums Shell Programming and Scripting Search multiple patterns in multiple files Post 302491078 by Chubler_XL on Wednesday 26th of January 2011 04:39:53 PM
Old 01-26-2011
Yesterday, I tried loading the keys into a c++ B-Tree and then pulling a key from your zip-string (11 chars from position 60) and doing a lookup. This managed to check 20,000 zip records in approx 0.7 sec. This highlights the power of indexed lookups, more work should probably be done in parsing your data records and determining the key from the record.


From the example zip record in my previous post we could lookup:
Code:
"56456456546"
"6456456546 "
"456456546  K"
"56456546 KK"
"6456546 KKK"
...
"MMMM0345353"
"MMM03453534"

Depending on the region where data appears and what limitation you have on key characters (eg are spaces or letters allowed in the keys), up to 27 lookups would be required for each record, this should still be able to process close to 1000 records per second.

As you can see it's important to find out as much about your data as possible, Ideally the processing program should look at a record and identify it's type and extract the proper key exactly and report errors on badly formatted records. This avoids false positives, ensures you are looking up what is intended and can react to format changes within the data records.

As methyl pointed out earlier getting a Systems Analyst involved is your best bet as we can only advise on information you supply here and there may be even better ways to get the result you require.
Last edited by Chubler_XL; 01-26-2011 at 06:00 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to parameterize multiple search patterns and generate a new file

I have one file: 123*100*abcd*10 123*101*abcd*-29*def 123*100*abcd*-10 123*102*abcd*-105*asd I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern. *100* and *- (ie *minus) *102* and *- The output that is... (6 Replies)
Discussion started by: augustinep
6 Replies

2. Shell Programming and Scripting

Perl: Match a line with multiple search patterns

Hi I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns. Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Discussion started by: Juha
10 Replies

3. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

4. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

5. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

6. Shell Programming and Scripting

Search & Replace: Multiple Strings / Multiple Files

I have a list of files all over a file system e.g. /home/1/foo/bar.x /www/sites/moose/foo.txtI'm looking for strings in these files and want to replace each occurrence with a replacement string, e.g. if I find: '#@!^\&@ in any of the files I want to replace it with: 655#@11, etc. There... (2 Replies)
Discussion started by: spacegoose
2 Replies

7. Shell Programming and Scripting

Grep from multiple patterns multiple file multiple output

Hi, I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command. Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns. Xargs -I {} grep... (3 Replies)
Discussion started by: Diya123
3 Replies

8. Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Hi All, I am starting a service which will redirect its out put into 2 logs say A and B. Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly. Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Discussion started by: Girish19
19 Replies

9. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies

10. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies

LEARN ABOUT SUSE

tracker-search

tracker-search(1)						   User Commands						 tracker-search(1)

NAME

       tracker-search - Search all content for keywords

SYNOPSIS

       tracker-search [OPTION...] EXPRESSION [EXPRESSION...]

DESCRIPTION

       tracker-search  searches all indexed content for EXPRESSION. The resource in which EXPRESSION matches must exist (see --all for more infor-
       mation). All results are returned in ascending order. In all cases, if no EXPRESSION is given for an argument (like --folders for  example)
       then ALL items in that category are returned instead.

       EXPRESSION
	      One or more terms to search. The default operation is a logical AND.  For logical OR operations, see -r.

OPTIONS

       -?, --help
	      Give a short help message.

       -l, --limit=N
	      Limit search to N results. The default is 512.

       -o, --offset=N
	      Offset the search results by N. For example, start at item number 10 in the results. The default is 0.

       -r, --or-operator
	      Use OR for search terms instead of AND (the default)

       -d, --detailed
	      Show the unique URN associated with each search result. This does not apply to --music-albums and --music-artists.

       -a, --all
	      Show  results  which might not be available. This might bebecause a removable media is not mounted for example. Without this option,
	      resources are only shown if they exist. This option applies to all command line switches except --music-albums and --music-artists.

       -f, --files=EXPRESSION
	      Search for files of any type matching EXPRESSION (optional).

       -e, --folders=EXPRESSION
	      Search for folders matching EXPRESSION (optional).

       -m, --music=EXPRESSION
	      Search for music files matching EXPRESSION (optional).

       --music-albums=ALBUM
	      Search for music albums matching ALBUM (optional).

       --music-artists=ARTIST
	      Search for music artists matching ARTIST (optional).

       -l, --images=EXPRESSION
	      Search for images matching EXPRESSION (optional).

       -v, --videos=EXPRESSION
	      Search for videos matching EXPRESSION (optional).

       -t, --documents=EXPRESSION
	      Search for documents matching EXPRESSION (optional).

       -e, --emails=EXPRESSION
	      Search for emails matching EXPRESSION (optional). Returns a list of subjects for emails found.

       -c, --contacts=EXPRESSION
	      Search for contacts matching EXPRESSION (optional). Returns a list of names and email addresses found.

       -V, --version
	      Print version.

SEE ALSO

       tracker-store(1), tracker-stats(1), tracker-tag(1), tracker-info(1).

GNU
								     July 2009							 tracker-search(1)
All times are GMT -4. The time now is 02:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy