Sponsored Content
Top Forums Shell Programming and Scripting In memory grep of a large file. Post 302960576 by Don Cragun on Tuesday 17th of November 2015 05:44:17 AM
Old 11-17-2015
The REs required to match two words of context before and after a specified word using grep are much more complex than the exact word matches I'm using in the awk field in array tests. So I was hoping it would be significantly faster, but without your actual data to use as a testbed there wasn't any way to be sure. I'm glad it is working well for you.
This User Gave Thanks to Don Cragun For This Post:
 

10 More Discussions You Might Find Interesting

1. Linux

shmat() Failure While Using a Large Amount of Shared Memory

Hi, I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size... (2 Replies)
Discussion started by: theicarusagenda
2 Replies

2. HP-UX

How can I get memory usage or anything that show memory used from sar file?

Refer from title: How can i get memory used or anything that can show memory from sar file example on solaris:- we can use sar with option to show memory used at time that sar crontab run. on HP-UX, it not has option to see memory used. But i think it may be have some parameter or some... (1 Reply)
Discussion started by: panithat
1 Replies

3. UNIX for Dummies Questions & Answers

Grep alternative to handle large numbers of files

I am looking for a file with 'MCR0000000716214' in it. I tried the following command: grep MCR0000000716214 * The problem is that the folder I am searching in has over 87000 files and I am getting the following: bash: /bin/grep: Arg list too long Is there any command I can use that can... (6 Replies)
Discussion started by: runnerpaul
6 Replies

4. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

5. Shell Programming and Scripting

grep error: range endpoint too large

Hi, my problem: gzgrep "^.\{376\}8301685001120" filename /dev/null ###ERROR ### grep: RE error 11: Range endpoint too large. Whats my mistake? Is the position 376 to large for grep??? Thanks. (2 Replies)
Discussion started by: Timmää
2 Replies

6. Shell Programming and Scripting

grep/fgrep/egrep for a very large matrix

All, I have a problem with grep/fgrep/egrep. Basically I am building a 200 times 200 correlation matrix. The entries of this matrix need to be retrieved from another very large matrix (~100G). I tried to use the grep/fgrep/egrep to locate each entry and put them into one file. It looks very... (1 Reply)
Discussion started by: realwindfly
1 Replies

7. UNIX for Advanced & Expert Users

Out of Memory error when free memory size is large

I was running a program and it stopped and showed "Out of Memory!". at that time, the RAM used by this process is around 4G and the free memory size of the machine is around 30G. Does anybody know what maybe the reason? this program is written with Perl. the OS of the machine is Solaris U8. And I... (1 Reply)
Discussion started by: lilili07
1 Replies

8. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

9. UNIX for Dummies Questions & Answers

virtual memory and diff'ing very large files

(0 Replies)
Discussion started by: uiop44
0 Replies

10. Shell Programming and Scripting

Large search replace using sed results in memory problem.

I have one big file of size 9GB (big_file.txt). This big file has sentences and paragraphs like any usual English document. I have another file consisting of replacement strings for sed to use. The file name is replace.sed and each entry in one line looks like this: s/\<shout\>/shout/g s/\<b is... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies
GIT-GREP(1)							    Git Manual							       GIT-GREP(1)

NAME
git-grep - Print lines matching a pattern SYNOPSIS
git grep [-a | --text] [-I] [-i | --ignore-case] [-w | --word-regexp] [-v | --invert-match] [-h|-H] [--full-name] [-E | --extended-regexp] [-G | --basic-regexp] [-F | --fixed-strings] [-n] [-l | --files-with-matches] [-L | --files-without-match] [-z | --null] [-c | --count] [--all-match] [-q | --quiet] [--max-depth <depth>] [--color[=<when>] | --no-color] [-A <post-context>] [-B <pre-context>] [-C <context>] [-f <file>] [-e] <pattern> [--and|--or|--not|(|)|-e <pattern>...] [--cached | --no-index | <tree>...] [--] [<pathspec>...] DESCRIPTION
Look for specified patterns in the tracked files in the work tree, blobs registered in the index file, or blobs in given tree objects. OPTIONS
--cached Instead of searching tracked files in the working tree, search blobs registered in the index file. --no-index Search files in the current directory, not just those tracked by git. -a, --text Process binary files as if they were text. -i, --ignore-case Ignore case differences between the patterns and the files. -I Don't match the pattern in binary files. --max-depth <depth> For each <pathspec> given on command line, descend at most <depth> levels of directories. A negative value means no limit. -w, --word-regexp Match the pattern only at word boundary (either begin at the beginning of a line, or preceded by a non-word character; end at the end of a line or followed by a non-word character). -v, --invert-match Select non-matching lines. -h, -H By default, the command shows the filename for each match. -h option is used to suppress this output. -H is there for completeness and does not do anything except it overrides -h given earlier on the command line. --full-name When run from a subdirectory, the command usually outputs paths relative to the current directory. This option forces paths to be output relative to the project top directory. -E, --extended-regexp, -G, --basic-regexp Use POSIX extended/basic regexp for patterns. Default is to use basic regexp. -F, --fixed-strings Use fixed strings for patterns (don't interpret pattern as a regex). -n Prefix the line number to matching lines. -l, --files-with-matches, --name-only, -L, --files-without-match Instead of showing every matched line, show only the names of files that contain (or do not contain) matches. For better compatibility with git diff, --name-only is a synonym for --files-with-matches. -z, --null Output instead of the character that normally follows a file name. -c, --count Instead of showing every matched line, show the number of lines that match. --color[=<when>] Show colored matches. The value must be always (the default), never, or auto. --no-color Turn off match highlighting, even when the configuration file gives the default to color output. Same as --color=never. -[ABC] <context> Show context trailing (A -- after), or leading (B -- before), or both (C -- context) lines, and place a line containing -- between contiguous groups of matches. -<num> A shortcut for specifying -C<num>. -p, --show-function Show the preceding line that contains the function name of the match, unless the matching line is a function name itself. The name is determined in the same way as git diff works out patch hunk headers (see Defining a custom hunk-header in gitattributes(5)). -f <file> Read patterns from <file>, one per line. -e The next parameter is the pattern. This option has to be used for patterns starting with - and should be used in scripts passing user input to grep. Multiple patterns are combined by or. --and, --or, --not, ( ... ) Specify how multiple patterns are combined using Boolean expressions. --or is the default operator. --and has higher precedence than --or. -e has to be used for all patterns. --all-match When giving multiple pattern expressions combined with --or, this flag is specified to limit the match to files that have lines to match all of them. -q, --quiet Do not output matched lines; instead, exit with status 0 when there is a match and with non-zero status when there isn't. <tree>... Instead of searching tracked files in the working tree, search blobs in the given trees. -- Signals the end of options; the rest of the parameters are <pathspec> limiters. <pathspec>... If given, limit the search to paths matching at least one pattern. Both leading paths match and glob(7) patterns are supported. EXAMPLES
git grep time_t -- *.[ch] Looks for time_t in all tracked .c and .h files in the working directory and its subdirectories. git grep -e '#define' --and ( -e MAX_PATH -e PATH_MAX ) Looks for a line that has #define and either MAX_PATH or PATH_MAX. git grep --all-match -e NODE -e Unexpected Looks for a line that has NODE or Unexpected in files that have lines that match both. AUTHOR
Originally written by Linus Torvalds <torvalds@osdl.org[1]>, later revamped by Junio C Hamano. DOCUMENTATION
Documentation by Junio C Hamano and the git-list <git@vger.kernel.org[2]>. GIT
Part of the git(1) suite NOTES
1. torvalds@osdl.org mailto:torvalds@osdl.org 2. git@vger.kernel.org mailto:git@vger.kernel.org Git 1.7.1 07/05/2010 GIT-GREP(1)
All times are GMT -4. The time now is 10:21 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy