Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Find records with matching patterns Post 302577472 by sbhuvana20 on Tuesday 29th of November 2011 06:33:03 AM
Old 11-29-2011
Find records with matching patterns

Hi,

I need to find records with a search string from a file. Search strings are provided in a file.

For eg. search_String.txt file is like below
chicago
mexico
newark
sanhose

and the file from where the records need to be fetched is given below

src_file:
xxx|chicago|12/31/2011|abc@xyz.com|installed
yyy|sanhose|12/30/2011|yyy@xyz.com|deleted
zzz|newark|11/30/2011|zzz@xyz.com|installed
ccc|denver|12/31/2011|ccc@xyz.com|installed


I have written a code as below using grep

#!/usr/bin/ksh
LIST=search_String.txt

while read line
do
grep ${line} src_file >> OUTPUT.TXT
done < ${LIST}

exit 0

This code works. But the problem am facing is my src_file is a very huge file having around 23 million records and I need to search for 1200 strings from this src_file and get the records from src_file.
Given this situation my code runs for hours.

Could anyone of you help me to understand if there is a better way for this search?

Thanks
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

matching 3 patterns in shell script

IN a file I need to check for 3 patterns if all the 3 patterns are in the file. I need to send out an email. All this needs to be done in korn shell script. Please advise. (1 Reply)
Discussion started by: saibsk
1 Replies

2. Shell Programming and Scripting

AWK: matching patterns in 2 different files

In a directory, there are two different file extensions (*.txt and *.xyz) having similar names of numerical strings (*). The (*.txt) contains 5000 multiple files and the (*.xyz) also contains 5000 multiple files. Each of the files has around 4000 rows and 8 columns, with several unique string... (5 Replies)
Discussion started by: asanjuan
5 Replies

3. Shell Programming and Scripting

Matching patterns

I have a file name in $f. If $f has "-" at the beginning, or "=", or does not have extension ".ry" or ".xt" or ".dat" then cerr would not be empty. Tried the following but having some problems. set cerr = `echo $f | awk '/^-|=|!.ry|!.xt|!.dat/'` (4 Replies)
Discussion started by: kristinu
4 Replies

4. Shell Programming and Scripting

how to form Records[multiple line] between two known patterns

file contents looks like this : #START line1 of record1 line2 of record1 #END #START line1 of record2 line2 of record2 line3 of record2 #END #START line1 of record3 #END my question how should i make it a records between #START and #END . willl i be able to get the contents of the... (5 Replies)
Discussion started by: sathish92
5 Replies

5. Shell Programming and Scripting

Find matched patterns and print them with other patterns not the whole line

Hi, I am trying to extract some patterns from a line. The input file is space delimited and i could not use column to get value after "IN" or "OUT" patterns as there could be multiple white spaces before the next digits that i need to print in the output file . I need to print 3 patterns in a... (3 Replies)
Discussion started by: redse171
3 Replies

6. Shell Programming and Scripting

Finding matching patterns in two files

Hi, I have requirement to find the matching patterns of two files in Unix. One file is the log file and the other is the error list file. If any pattern in the log file matches the list of errors in the error list file, then I would need to find the counts of the match. For example, ... (5 Replies)
Discussion started by: Bobby_2000
5 Replies

7. Shell Programming and Scripting

Find files not matching multiple patterns and then delete anything older than 10 days

Hi, I have multiple files in my log folder. e.g: a_m1.log b_1.log c_1.log d_1.log b_2.log c_2.log d_2.log e_m1.log a_m2.log e_m2.log I need to keep latest 10 instances of each file. I can write multiple find commands but looking if it is possible in one line. m file are monthly... (4 Replies)
Discussion started by: wahi80
4 Replies

8. Shell Programming and Scripting

Bash - Find files excluding file patterns and subfolder patterns

Hello. For a given folder, I want to select any files find $PATH1 -f \( -name "*" but omit any files like pattern name ! -iname "*.jpg" ! -iname "*.xsession*" ..... \) and also omit any subfolder like pattern name -type d \( -name "/etc/gconf/gconf.*" -o -name "*cache*" -o -name "*Cache*" -o... (2 Replies)
Discussion started by: jcdole
2 Replies

9. Shell Programming and Scripting

Delete patterns matching

Delete patterns matching OS version: RHEL 7.3 Shell : Bash I have a file like below (pattern.txt). I need to delete all lines starting with the following words (words separated by comma below) and ) character. LOGGING, NOCOMPRESS, TABLESPACE , PCTFREE, INITRANS, MAXTRANS, STORAGE,... (3 Replies)
Discussion started by: John K
3 Replies

10. Shell Programming and Scripting

Extended grep not matching some patterns

i have a file where the hostnames and variables are in same line in below format, am able extract some part variables while otherlike subscriptions and handler is missing. can you please correct me if grep is able to perform this ? cat /tmp/test localhost subscriptions='' handler="genie"... (14 Replies)
Discussion started by: rakeshkumar
14 Replies
iofind(1M)																iofind(1M)

NAME
iofind - find and convert ASCII files containing the legacy representation of device (special file name or hardware path) SYNOPSIS
file] directory] filelist] file] directory] filelist] DESCRIPTION
The command helps users with the migration to the agile naming model (see intro(7)). First, identifies ASCII files on the system that con- tain patterns representing legacy mass storage device special file names or hardware paths. Then replaces the matching patterns with their mapping in the agile naming model inside the ASCII files. NOTE: may take several minutes to execute depending on the number of files to be searched (see the options). If there are many files, the search will take a long time. An alternative is to redirect the output to a file and run in background mode (for example, The increasing size of this output file indicates that is making progress. While executing, the command creates the following files in the and directories. Files under the directory: o contains the list of valid legacy device special file patterns to search for. o contains the list of valid legacy hardware path patterns to search for. o contains a colon separated list of all legacy device special files mapped to their corresponding persistent device special files. The first field is the persistent device special file name and the second field is the legacy name. o contains a colon separated list of all legacy hardware paths mapped to their corresponding lun and lunpath hardware paths in the agile naming model. The first and second fields are the lun and lunpath hardware paths, respectively, in the agile naming model. The third field is the legacy path. o contains a list of files which match at least one of the device special file search patterns in o contains a list of files which match at least one of the hardware path search patterns in o contains the list of files to be searched. o contains the command output summary, which is also displayed on the console during execution. File under the directory: o contains a list of directories to be excluded from the search. The purpose of this file is to speed up the search by not searching directories indicated in this file. Running the iofind command will create this file if it does not already exist. If a user provides the option, this file is not used to exclude directories. A user can edit this file to add or delete any directories from the search. Delete all the entries in the file if you do not want iofind to skip any directories. By default, the following directories are repre- sented in this file: /var/adm/msgbuf/ /var/adm/iofind/ /var/adm/syslog/ /var/adm/crash/ /var/adm/userdb/ /var/opt/ignite/ /usr/share/lib/ /usr/share/man/ /var/spool/ /etc/opt/resmon/ /opt/ignite/boot/ /opt/hpsmh/ /opt/openssl/ /opt/ssh/ /opt/wbem/ /opt/atok/ /opt/X11/ /opt/java/ /opt/gnome /opt/java /opt/mozilla /opt/wrkio /opt/msscaf/ /opt/aCC/ /opt/sfmdb/ /usr/lib/ /usr/bin/ /usr/dt/ /usr/include/ /var/evm/ /var/stm/ /stand/ /tmp/ Files under the directory: o These are backup copies of the original files, if any, before they get modified by the -R option. They are stored including their parent directory. For instance, the file /usr/mydata will be saved under Files under the directory: o When option is used along with the original files are not actually modified, but a preview of the modified files, including their parent directory, is made available under this preview directory. If a user does not use the option to specify a file or files to search, the iofind command will search through directories, except those indicated in the file, and it will skip files with the following file extensions: .ttf .gif .jpg .jar .html .img .xml .mof .so.1 .so .sdl .png .css .tar .gz .zip .1m .pl The option can be used to specify a list of patterns to be searched in ASCII files on the system. The patterns to be searched must be entered in the input file, file, one per line. This input file must contain legacy device special files, legacy hardware paths, or a mix of both. Device special file names must be fully qualified. For example, the full path must be used instead of If is invoked with the option, and no input file is specified using the option, builds a list of legacy device special file patterns from the data returned by the command. If is invoked with the option, and no input file is specified using the option, builds a list of legacy hardware path patterns by calling the command. The and options can be used together to search both types of patterns. If is invoked with the option, it replaces the matching patterns with their mapping in all ASCII files identified, unless the (preview) option is specified. Before modifying the files, saves a copy of the original ASCII files in the directory. also asks for confirmation before each change, unless the (force) option is used. Options recognizes the following options: Specify the directory which contains the files to be searched. If the option is not specified, then the current directory is used. All directories under the specified directory are searched recursively. Turn off interactive mode (also known as the force option). This option must be used with the option. The user is not prompted to confirm the replacement of a pattern before the pattern is replaced. Specify a list of patterns in the input file to search for. The format of the input file is one device special file or hardware path, per line. When an input file is provided using the option, the user must also specify the or option. Search ASCII files on the system for legacy hardware path patterns. See "NOTE" in about displaying output to a file. If the option is not specified, first calls to generate a list of hardware path patterns to be searched for. Otherwise, if the option is specified, searches the ASCII files for the hardware path patterns defined in the file. Patterns are saved in the file. Either the or option must be specified. Also, both options can be specified together. Print the command usage. When is invoked with no command line options, or with the option, it prints a usage message. Provide a list of files under the specified directory, for which to search for legacy device special files and hardware paths. The user may list as many files as needed on the com- mand line when using the option. Each file must be separated by a comma. Search ASCII files on the system for legacy device special file patterns. See "NOTE" in about displaying output to a file. If the option is not specified, first calls to generate a list of device special file patterns to be searched for. Otherwise, if the option is specified, searches the ASCII files for the devices special file patterns defined in the file. Patterns are saved in the file. Either the or option must be specified. Also, both options can be specified together. Preview, but do not replace, the selected patterns. This option is only supported in combination with the option. When both the and options are specified, does not do the actual replacement of matching patterns (see the option), but the replacement information will be created under this directory: Replace matching patterns with their mapping in the agile naming model. If the option is used with the option, a preview of changes will be provided in the directory. If the option is used without the option, asks the user for a confirmation before executing each replacement in the file. If used with the option, the replacement is executed without asking the user for confirmation. You must be superuser to execute the option. You do not have to be superuser to execute the other options. All files modified by this command are first backed up in the directory. RETURN VALUE
Exit values are: Successful completion. An error condition occurred. EXAMPLES
Generate a list of searchable hardware paths on the system. Generate a list of searchable hardware paths on the system. Redirect the output to a file and run as a background task. Find all patterns matching the legacy hardware paths, 0/1/1/0.0.0 and 1/2/0/0.3.0, in the file. This file contains the strings, 0/1/1/0.0.0 and 1/2/0/0.3.0, one string per line. Search all files under the current directory for matching hardware path patterns found in and preview the replacement of any matching pat- terns without doing the actual replacement in the files. Search all files under the current directory for matching device special file patterns found in and replace all matching patterns with their mapping in the agile naming model. The format of the input file is one full path device special file, such as /dev/dsk/c1t3d0, per line. Do not prompt the user for confirmation before each replacement. Scan through all files under and its subdirectories to find all patterns matching the legacy hardware paths, 0/1/1/0.0.0 and 1/2/0/0.3.0, in the file. This file contains the strings, 0/1/1/0.0.0 and 1/2/0/0.3.0, one string per line. WARNINGS
The replace functionality needs appropriate permissions to make changes to the files to be modified. SEE ALSO
ioscan(1M), intro(7). iofind(1M)
All times are GMT -4. The time now is 02:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy