Sponsored Content
Top Forums UNIX for Advanced & Expert Users Best way to search for patterns in huge text files Post 302385624 by andy2000 on Friday 8th of January 2010 05:27:04 PM
Old 01-08-2010
thank u for your quick replay...but it does not work ..

I need something faster than the following solution:
Code:
for i in `cat pattern_file`
do
  grep $i  input_file1 >> output_file1
  grep $i  input_file2 >> output_file2  
  grep $i  input_file3 >> output_file3
done

Any ideas please!!!

Last edited by Scott; 01-08-2010 at 06:32 PM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. Solaris

Huge (repeated Entry) text files

Somebody HELP! I have a huge log file (TEXT) 76298035 bytes. It's a logfile of IMEIs and IMSIS that I get from my EIR node. Here is how the contents of the file look like: 000000, 1 33016382000913 652020100423994 1 33016382002353 652020100430743 1 33017035101003 652020100441736... (4 Replies)
Discussion started by: axl
4 Replies

2. UNIX Desktop Questions & Answers

how to search files efficiently using patterns

hi friens, :) if i need to find files with extension .c++,.C++,.cpp,.Cpp,.CPp,.cPP,.CpP,.cpP,.c,.C wat is the pattern for finding them :confused: (2 Replies)
Discussion started by: arunsubbhian
2 Replies

3. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

4. UNIX for Dummies Questions & Answers

script to search patterns inside list of files

>testfile while read x do if then echo $x >> testfile else fi if then echo $x >> testfile else fi done < list_of_files is there any efficient way to search abc.dml and xyz.dml ? (2 Replies)
Discussion started by: dr46014
2 Replies

5. Shell Programming and Scripting

to read two files, search for patterns and store the output in third file

hello i have two files temp.txt and temp_unique.text the second file consists the unique fields from the temp.txt file the strings stored are in the following form 4,4 17,12 15,65 4,4 14,41 15,65 65,89 1254,1298i'm able to run the following script to get the total count of a... (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

6. SuSE

Search all files based on first and in all listed files search the second patterns

Hello Linux Masters, I am not a linux expert therefore i need help from linux gurus. Well i have a requirement where i need to search all files based on first patterns and after seraching all files then serach second pattern in all files which i have extracted based on first pattern.... (1 Reply)
Discussion started by: Black-Linux
1 Replies

7. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

8. Shell Programming and Scripting

How to fix line breaks format text for huge files?

Hi, I need to correct line breaks for huge files (more than 1MM records in a file) and then format it properly. Except the header and trailer, each record starts with 'D'. Requirement:Scan the whole file except the header and trailer records and see if any of the records start with... (19 Replies)
Discussion started by: kikionline
19 Replies

9. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate... (17 Replies)
Discussion started by: danish0909
17 Replies

10. Shell Programming and Scripting

Search and replace ---A huge number of files

Hello Friends, I have the below scenario in my current project. Suggest me which tool ( perl,python etc) is best to this scenario. Or should I go for Programming language ( C/Java ).. (1) I will be having a very big file ( information about 200million subscribers will be stored in it ). This... (5 Replies)
Discussion started by: panyam
5 Replies
strextract(1int)														  strextract(1int)

Name
       strextract - batch string extraction

Syntax
       strextract [ -p patternfile ] [ -i ignorefile ] [ -d ] [ source-program...  ]

Description
       The  command extracts text strings from source programs.  This command also writes the string it extracts to a message text file.  The mes-
       sage text file contains the text for each message extracted from your input source program. The command names the file by appending .msg to
       the name of the input source program.

       In  the	source-program argument, you name one or more source programs from which you want messages extracted. The command does not extract
       messages from source programs included using the directive. Therefore, you might want a source program  and  all  the  source  programs	it
       includes on a single command line.

       You  can  create a patterns file (as specified by patternfile ) to control how the command extracts text. The patterns file is divided into
       several sections, each of which is identified by a keyword. The keyword must start at the beginning of a new line, and its first  character
       must  be  a dollar sign ($).  Following the identifier, you specify a number of patterns. Each pattern begins on a new line and follows the
       regular expression syntax you use in the routine. For more information on the patterns file, see the(5int) reference page.

       In addition to the patterns file, you can create a file that indicates strings that ignores.  Each line in this ignore file contains a sin-
       gle string to be ignored that follows the syntax of the routine.

       When you invoke the command, it reads the patterns file and the file that contains strings it ignores.  You can specify a patterns file and
       an ignore file on the command line.  Otherwise, the command matches all strings and uses the default patterns file.

       If finds strings which match the directive in the pattern file, it reports the strings to standard error (stderr.) but does not	write  the
       string to the message file.

       After running you can edit the message text file to remove text strings which do not need translating before running

       It is  recommended that you use command	as a visual front end to the command rather than running directly.

Options
       -i   Ignore  text  strings specified in ignorefile.  By default, the command searches for ignorefile in the current working directory, your
	    home directory, and

	    If you omit the option, recognizes all strings specified in the patterns file.

       -p   Use patternfile to match strings in the input source program. By default, the command searches for the pattern  file  in  the  current
	    working directory, your home directory, and finally

	    If you omit the option, the command uses a default patterns file that is stored in

       -d   Disables warnings of duplicate strings. If you omit the option, prints warnings of duplicate strings in your source program.

Restrictions
       Given the default pattern file, you cannot cause to ignore strings in comments that are longer than one line.

       You can specify only one rewrite string for all classes of pattern matches.

       The command does not extract strings from files include with directive. You must run the commands on these files separately.
       % strextract -p c_patterns prog.c prog2.c
       % vi prog.msg
       % strmerge -p c_patterns prog.c prog2.c
       % gencat prog.cat prog.msf prog2.msf
       % vi nl_prog.c
       % vi nl_prog2.c
       % cc nl_prog.c nl_prog2.c -li

       In this example, the command uses the file to determine which strings to match. The input source programs are named and

       If  you	need  to  remove  any  of the messages or extract one of the created strings, edit the resulting message file, Under no conditions
       should you add to this file. Doing so could result in unpredictable behavior.

       You issue the command to replace the extracted strings with calls to the message catalog.  In response to this command, creates the  source
       message catalogs, and and the output source programs, and

       You must edit and to include the appropriate and function calls.

       The command creates a message catalog and the command creates an executable program.

See Also
       intro(3int), gencat(1int), extract(1int), strmerge(1int), regex(3), catopen(3int), patterns(5int)
       Guide to Developing International Software

																  strextract(1int)
All times are GMT -4. The time now is 05:37 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy