Sponsored Content
Full Discussion: Avoiding external utilities
Top Forums Shell Programming and Scripting Avoiding external utilities Post 303010841 by rovf on Monday 8th of January 2018 08:35:57 AM
Old 01-08-2018
Quote:
Originally Posted by Don Cragun
Hi rovf,
On every system I've seen, for a large amount of data (which one might assume from a variable named MASSIVETEXT), there is a noticeable difference in performance between grep -F (fastest), grep without -E and without -F (slower), and grep -E (slower still).
Interesting point. I tried it with a 300MB file, using Cygwin grep, and searching for a fixed string with various settings. I repeated each run 3 times. Here the times

with -F:

Code:
0.27s user 0.08s system 91% cpu 0.372 total

0.30s user 0.05s system 93% cpu 0.367 total

0.28s user 0.06s system 95% cpu 0.358 total

without option:

Code:
0.22s user 0.14s system 98% cpu 0.362 total

0.31s user 0.05s system 98% cpu 0.363 total

0.22s user 0.12s system 95% cpu 0.358 total

with -E:

Code:
0.25s user 0.09s system 93% cpu 0.367 total

0.27s user 0.08s system 95% cpu 0.359 total

0.25s user 0.11s system 98% cpu 0.365 total

Of course, it could be that the amount of data is not massive enough to show a significant difference; or that the I/O of the Cygwin layer is so heavy, that it shadows performance differences. Or that gnu grep is clever enough to recognize, that the pattern does not contain any regexp characters and does internally a '-F' always.

To evaluate at the last hypothesis, I replaced the pattern by one which really contained an extended regexp instead of a fixed string (using a .+ regexp operation). In this case, the times were systematically higher, but not very much:

Code:
0.37s user 0.11s system 99% cpu 0.486 total

0.36s user 0.09s system 93% cpu 0.481 total

0.30s user 0.14s system 89% cpu 0.486 total

Perhaps such a test should really be repeated with a file which is several GB in size.

Interestingly, in the OP's problem, the whole content of the file was stored in a shell variable, and this makes me wonder, where the practical limit is for the content of a variable in, say, bash or ksh...

Last edited by rbatte1; 01-08-2018 at 10:49 AM.. Reason: Added CODE tags for output
These 2 Users Gave Thanks to rovf For This Post:
 

9 More Discussions You Might Find Interesting

1. SCO

Avoiding duplicates with some special case

Hi Gurus, I had a question regarding avoiding duplicates.i have a file abc.txt abc.txt ------- READER_1_1_1> HIER_28056 XML Reader: Error occurred while parsing:; line number ; column number READER_1_3_1> Sun Mar 23 23:52:48 2008 READER_1_3_1> HIER_28056 XML Reader: Error occurred while... (0 Replies)
Discussion started by: pssandeep
0 Replies

2. OS X (Apple)

Installing applications by avoiding GUI

Hi Experts, Now the problem is when I run the install script inside .app folder, it opens a GUI and asks for user input. I want to avoid these GUI. I want to provide input when i run install script e.g. $ ./install < inputfile I used to redirect input from input file to install script. But... (4 Replies)
Discussion started by: akash.mahakode
4 Replies

3. Shell Programming and Scripting

using find but avoiding sparse files

I am no Unix administrator...I live in windows land. I wrote a script to find files of certain names and process them but was later advised to avoid checking sparse files since it would use up a lot of resources and the files I was looking for were not there. How do I avoid doing the find on... (3 Replies)
Discussion started by: shellFun
3 Replies

4. UNIX for Dummies Questions & Answers

Avoiding the second run of the script

Hi all, I want to put a check in my script to check if the same instance is already running and not finished and if not then does not allow it to run! in which part of my script I should put this? and any idea how I should write it? tx (4 Replies)
Discussion started by: messi777
4 Replies

5. Shell Programming and Scripting

Avoiding 'sh -c' when running ps from CRON

Hi, I have a script which has the below line: ps -ef | grep ${SCRIPT_NAME} | grep ksh | grep -v grep >> /tmp/instance.tmp When the script is invoked through CRON, I get 2 lines in instance.tmp when actually only one instance is running: cdrd 17790 17789 0 15:14:01 ? 0:00 /bin/ksh... (8 Replies)
Discussion started by: cavallino4u
8 Replies

6. UNIX for Dummies Questions & Answers

Avoiding the history

In bash shell, how we can avoid the commands getting recorded in history file. One way i can think of is : export HISTSIZE=0 Is there any other way to achieve this? Thanks (1 Reply)
Discussion started by: pandeesh
1 Replies

7. Shell Programming and Scripting

Avoiding some files inside a loop

In my script I need to loop around some files like below example files are fa.info.abcd fa.info.bxde fa.info.cdas ------ for test_data in fa.info.* do # Some text appending logic applied # Copy to another directory done Now I need to discard some files while looping around ... (9 Replies)
Discussion started by: smile689
9 Replies

8. Shell Programming and Scripting

Search avoiding special characters

Hi all, I have a list which I want to search in another file. I can do that using grep -f but the search is failing due to special characters, how do I solve this? One row in that list is amino-acid permease inda1 gb|EDU41782.1| amino-acid permease inda1 Input file to be searched... (2 Replies)
Discussion started by: gina.lizar
2 Replies

9. Shell Programming and Scripting

Avoiding new line for the counts

Hi Team, Am getting the below output but need the count of records to be displayed in same line but currently count alone moves to next line. Please let me know how we can still keep the count in the same line. ######code ##### while read YEAR; do for i in TEST_*PGYR${YEAR}_${DT}.csv; do... (3 Replies)
Discussion started by: weknowd
3 Replies
All times are GMT -4. The time now is 05:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy