Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Speeding up a Shell Script (find, grep and a for loop) Post 302222037 by era on Wednesday 6th of August 2008 01:00:26 AM
Old 08-06-2008
I don't understand the construction set | grep $value /tmp/date.out -- as far as I can tell, the output from set will not be used for anything.

Also, the conditional is a Useless Use of Test, and it will print to stdout any matches; I imagine that's undesirable. The following avoids those problems.

Code:
if ! grep "$value" /tmp/dave.out >/dev/null
then
  echo "$value"" >>/tmp/dave_files.out
fi

Still, if you had your list of PDF files in another file, one PDF per line, it could be as simple as

Code:
fgrep -vxf pdfs.txt /tmp/dave.out >/tmp/dave_files.out

The use of fgrep -x requires an exact match (not a regex match; you know that dot in a regex matches any character, for example) spanning the whole line (that's the -x). The -v causes only lines in /tmp/dave.out which are not anywhere in pdfs.txt to be printed.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

2. Shell Programming and Scripting

Bash script (using find and grep)

I'm trying to make a simple search script but cannot get it right. The script should search for keywords inside files. Then return the file paths in a variable. (Each file path separated with \n). #!/bin/bash SEARCHQUERY="searchword1 searchword2 searchword3"; for WORD in $SEARCHQUERY do ... (6 Replies)
Discussion started by: limmer
6 Replies

3. Shell Programming and Scripting

Shell script / Grep / Awk to variable and Loop

Hi, I have a text file with data in that I wish to extract, assign to a variable and process through a loop. Kind of the process that I am after: 1: Grep the text file for the values. Currently using: cat /root/test.txt | grep TESTING= | awk -F"=" '{ a = $2 } {print a}' | sort -u ... (0 Replies)
Discussion started by: Spoonless
0 Replies

4. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

5. Shell Programming and Scripting

Speeding up search and replace in a for loop

Hello, I am using sed in a for loop to replace text in a 100MB file. I have about 55,000 entries to convert in a csv file with two entries per line. The following script works to search file.txt for the first field from conversion.csv and then replace it with the second field. While it works fine,... (15 Replies)
Discussion started by: pbluescript
15 Replies

6. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies

7. Shell Programming and Scripting

How to use grep in a loop using a bash script?

Dear all, Please help with the following. I have a file, let's call it data.txt, that has 3 columns and approx 700,000 lines, and looks like this: rs1234 A C rs1236 T G rs2345 G T Please use code tags as required by forum rules! I have a second file, called reference.txt,... (1 Reply)
Discussion started by: aberg
1 Replies

8. Shell Programming and Scripting

Speeding up shell script with grep

HI Guys hoping some one can help I have two files on both containing uk phone numbers master is a file which has been collated over a few years ad currently contains around 4 million numbers new is a file which also contains 4 million number i need to split new nto two separate files... (4 Replies)
Discussion started by: dunryc
4 Replies

9. Shell Programming and Scripting

Help 'speeding' up this 'parsing' script - taking 24+ hours to run

Hi, I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies

10. Shell Programming and Scripting

Help with speeding up my working script to take less time - how to use more CPU usage for a script

Hello experts, we have input files with 700K lines each (one generated for every hour). and we need to convert them as below and move them to another directory once. Sample INPUT:- # cat test1 1559205600000,8474,NormalizedPortInfo,PctDiscards,0.0,Interface,BG-CTA-AX1.test.com,Vl111... (7 Replies)
Discussion started by: prvnrk
7 Replies
grep(1) 						      General Commands Manual							   grep(1)

Name
       grep, egrep, fgrep - search file for regular expression

Syntax
       grep [option...] expression [file...]

       egrep [option...] [expression] [file...]

       fgrep [option...] [strings] [file]

Description
       Commands  of  the family search the input files (standard input default) for lines matching a pattern.  Normally, each line found is copied
       to the standard output.

       The command patterns are limited regular expressions in the style of which uses a compact nondeterministic algorithm.  The command patterns
       are  full  regular  expressions.  The command uses a fast deterministic algorithm that sometimes needs exponential space.  The command pat-
       terns are fixed strings.  The command is fast and compact.

       In all cases the file name is shown if there is more than one input file.  Take care when using the characters $ * [ ^ | ( ) and   in  the
       expression because they are also meaningful to the Shell.  It is safest to enclose the entire expression argument in single quotes ' '.

       The command searches for lines that contain one of the (new line-separated) strings.

       The command accepts extended regular expressions.  In the following description `character' excludes new line:

	      A  followed by a single character other than new line matches that character.

	      The character ^ matches the beginning of a line.

	      The character $ matches the end of a line.

	      A .  (dot) matches any character.

	      A single character not otherwise endowed with special meaning matches that character.

	      A  string  enclosed in brackets [] matches any single character from the string.	Ranges of ASCII character codes may be abbreviated
	      as in `a-z0-9'.  A ] may occur only as the first character of the string.  A literal - must be placed where it can't be mistaken	as
	      a range indicator.

	      A  regular  expression  followed	by  an	* (asterisk) matches a sequence of 0 or more matches of the regular expression.  A regular
	      expression followed by a + (plus) matches a sequence of 1 or more matches of the regular expression.  A regular expression  followed
	      by a ? (question mark) matches a sequence of 0 or 1 matches of the regular expression.

	      Two regular expressions concatenated match a match of the first followed by a match of the second.

	      Two regular expressions separated by | or new line match either a match for the first or a match for the second.

	      A regular expression enclosed in parentheses matches a match for the regular expression.

       The  order  of  precedence  of  operators at the same parenthesis level is the following:  [], then *+?, then concatenation, then | and new
       line.

Options
       -b	   Precedes each output line with its block number.  This is sometimes useful in locating disk block numbers by context.

       -c	   Produces count of matching lines only.

       -e expression
		   Uses next argument as expression that begins with a minus (-).

       -f file	   Takes regular expression (egrep) or string list (fgrep) from file.

       -i	   Considers upper and lowercase letter identical in making comparisons and only).

       -l	   Lists files with matching lines only once, separated by a new line.

       -n	   Precedes each matching line with its line number.

       -s	   Silent mode and nothing is printed (except error messages).	This is useful for checking the error status (see DIAGNOSTICS).

       -v	   Displays all lines that do not match specified expression.

       -w	   Searches for an expression as for a word (as if surrounded by `<' and `>').  For further information, see only.

       -x	   Prints exact lines matched in their entirety only).

Restrictions
       Lines are limited to 256 characters; longer lines are truncated.

Diagnostics
       Exit status is 0 if any matches are found, 1 if none, 2 for syntax errors or inaccessible files.

See Also
       ex(1), sed(1), sh(1)

																	   grep(1)
All times are GMT -4. The time now is 04:59 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy