Sponsored Content
Top Forums Shell Programming and Scripting grep/fgrep/egrep for a very large matrix Post 302423108 by jim mcnamara on Thursday 20th of May 2010 07:37:59 AM
Old 05-20-2010
simply reading 100GB is going to take several minutes, so you should get everything you need on one pass thru the file. If you read through the entire file 200 times you are doomed to wait for looong time.

Actually, you can "tune" a regex. You can also use awk with a regex. awk can output simultaneously to OPEN_MAX differentt files.
Code:
 getconf OPEN_MAX

show us what your input is like and what you want as output.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

I need help with fgrep or grep

How can I do an and condition with fgrep. I want to do: ps -ef | fgrep -f searchvalues > tempmail.file mailx -s "Email Subject" email@domain.com < tempmail.file The search values file contains: opt/bea.*java.*80 mysqld What I want is to find things that contain: mysqld OR... (7 Replies)
Discussion started by: jimmy
7 Replies

2. UNIX Desktop Questions & Answers

Difference grep, egrep and fgrep

Hi All, Can anyone please explain me the difference between grep, egrep and fgrep with examples. I am new to unix environment.. Your help is highly appreciated. Regards, ravi (2 Replies)
Discussion started by: ravind27
2 Replies

3. UNIX for Dummies Questions & Answers

Difference between grep, egrep & grep -i

Hi All, Please i need to know the difference between grep, egrep & grep -i when used to serach through a file. My platform is SunOS 5.9 & i'm using the korn shell. Regards, - divroro12 - (2 Replies)
Discussion started by: divroro12
2 Replies

4. Shell Programming and Scripting

Fgrep or grep or awk help - scanning for delimiters.

Hi, I'm struggling a little here, so I figured it's time to ask for help. I have a file with a list of several hundred IDs (the hit file- "hitfile.txt"), which is newline delimited, and a much bigger (~500Mb) text file, "FASTA.txt" with several thousand entries, delimited by ">". It's the... (8 Replies)
Discussion started by: Tbox
8 Replies

5. Shell Programming and Scripting

Using grep or egrep

So a few months ago, I decided to move away from using grep and decided to use egrep in this code that i'm writing. i figured egrep is more robust than grep. well, it appears it isn't. when i used egrep to search the log file for a script that looked like the following, egrep couldn't find... (3 Replies)
Discussion started by: SkySmart
3 Replies

6. Homework & Coursework Questions

grep and egrep

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I have a text file. 1) How do I search for x and y? 'Find all lines that contain David and Emily' 2) How do I... (1 Reply)
Discussion started by: ninjagod123
1 Replies

7. Shell Programming and Scripting

help printing two consecutive columns, every twenty in a large matrix

Hi, I'm having a problem printing two consecutive columns, as I iterate through a large matrix by twenty columns and I was looking for a solution. My input file looks something like this 1 id1 A1 A2 A3 A4 A5 A6....A20 A21 A22 A23....A4001 A4002 2 id2 B1 B2 B3 B4 B5 B6... 3 id3 ... 4 id4... (8 Replies)
Discussion started by: flotsam
8 Replies

8. Shell Programming and Scripting

Fgrep/grep -f and literal strings

I have a file like this: cat file name = server jobname = 1010 snapshot_name = funky_Win2k12_20140213210409 I'm trying to use grep to isolate that first line (name = server), but grep -f "name = " file as well as fgrep "name = " file returns all 3 lines. How do I return... (1 Reply)
Discussion started by: ampsys
1 Replies

9. UNIX for Advanced & Expert Users

Need to use fgrep and egrep in a single command

I have a source file which is something like : C/*M/ / ***HMACCT ** MONTH FOR CURRENT MINUS 14 CAL DAY C/*D/ / ***HMACCT ** DAY FOR CURRENT MINUS 14 CAL DAY C/*X/ / ***HMACCT ** CENTURY FOR CURRENT MINUS 14 CAL DAY C/*Y/ / ***HMACCT **... (5 Replies)
Discussion started by: gotamp
5 Replies

10. Shell Programming and Scripting

Parsing a subset of data from a large matrix

I do have a large matrix of the following format and it is tab delimited ch-ab1-20 ch-bb2-23 ch-ab1-34 ch-ab1-24 er-cc1-45 bv-cc1-78 ch-ab1-20 0 2 3 4 5 6 ch-bb2-23 3 0 5 ... (6 Replies)
Discussion started by: Kanja
6 Replies
regex(1F)							   FMLI Commands							 regex(1F)

NAME
regex - match patterns against a string SYNOPSIS
regex [-e] [ -v "string"] [ pattern template] ... pattern [template] DESCRIPTION
The regex command takes a string from the standard input, and a list of pattern / template pairs, and runs regex() to compare the string against each pattern until there is a match. When a match occurs, regex writes the corresponding template to the standard output and returns TRUE. The last (or only) pattern does not need a template. If that is the pattern that matches the string, the function simply returns TRUE. If no match is found, regex returns FALSE. The argument pattern is a regular expression of the form described in regex(). In most cases, pattern should be enclosed in single quotes to turn off special meanings of characters. Note that only the final pattern in the list may lack a template. The argument template may contain the strings $m0 through $m9, which will be expanded to the part of pattern enclosed in ( ... )$0 through ( ... )$9 constructs (see examples below). Note that if you use this feature, you must be sure to enclose template in single quotes so that FMLI does not expand $m0 through $m9 at parse time. This feature gives regex much of the power of cut(1), paste(1), and grep(1), and some of the capabilities of sed(1). If there is no template, the default is $m0$m1$m2$m3$m4$m5$m6$m7$m8$m9. OPTIONS
The following options are supported: -e Evaluates the corresponding template and writes the result to the standard output. -v "string" Uses string instead of the standard input to match against patterns. EXAMPLES
Example 1: Cutting letters out of a string To cut the 4th through 8th letters out of a string (this example will output strin and return TRUE): `regex -v "my string is nice" '^.{3}(.{5})$0' '$m0'` Example 2: Validating input in a form In a form, to validate input to field 5 as an integer: valid=`regex -v "$F5" '^[0-9]+$'` Example 3: Translating an environment variable in a form In a form, to translate an environment variable which contains one of the numbers 1, 2, 3, 4, 5 to the letters a, b, c, d, e: value=`regex -v "$VAR1" 1 a 2 b 3 c 4 d 5 e '.*' 'Error'` Note the use of the pattern '.*' to mean "anything else". Example 4: Using backquoted expressions In the example below, all three lines constitute a single backquoted expression. This expression, by itself, could be put in a menu defini- tion file. Since backquoted expressions are expanded as they are parsed, and output from a backquoted expression (the cat command, in this example) becomes part of the definition file being parsed, this expression would read /etc/passwd and make a dynamic menu of all the login ids on the system. `cat /etc/passwd | regex '^([^:]*)$0.*$' ' name=$m0 action=`message "$m0 is a user"`'` DIAGNOSTICS
If none of the patterns match, regex returns FALSE, otherwise TRUE. NOTES
Patterns and templates must often be enclosed in single quotes to turn off the special meanings of characters. Especially if you use the $m0 through $m9 variables in the template, since FMLI will expand the variables (usually to "") before regex even sees them. Single characters in character classes (inside []) must be listed before character ranges, otherwise they will not be recognized. For exam- ple, [a-zA-Z_/] will not find underscores (_) or slashes (/), but [_/a-zA-Z] will. The regular expressions accepted by regcmp differ slightly from other utilities (that is, sed, grep, awk, ed, and so forth). regex with the -e option forces subsequent commands to be ignored. In other words, if a backquoted statement appears as follows: `regex -e ...; command1; command2` command1 and command2 would never be executed. However, dividing the expression into two: `regex -e ...``command1; command2` would yield the desired result. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
awk(1), cut(1), grep(1), paste(1), sed(1), regcmp(3C), attributes(5) SunOS 5.10 12 Jul 1999 regex(1F)
All times are GMT -4. The time now is 06:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy