Unexpected Results (at least I did not expect them)
I have two sripts running in bash. The first one uncompresses log files and moves them to a working directory using uncompress -c and > output to new directory. It then creates one control record to assure our search returns a record. It then calls or executes the second script, which is a grep for a literal string (ip and / or url).
My control record for this test was 251.251.251.251 and the data returned was my expected control record plus the following;
I am supposing that it matched on the 251,251,251,251. Here is my code;
Code:
#!/usr/bin/bash
#
# SearchLogsB
#
# SearchLogsB is called by SearchLogs and will look through the Search Files
# locating the Search Criteria requested.
#
# SEARCHDIR is the directory we created in Part A for doing our search work.
#
# SearchCriteria is the file holding the strings to search for. Normally these
# will be ip sets and urls.
#
# SearchFiles is the file holding the dates for the logs you are searching.
# Echo the info to the our status file so that we may be sure runs with a large
# number of Search Files are executing.
echo "grepping from $SearchFiles" >> $SEARCHDIR/search.status
echo "grepping from $SearchFiles" >> $SEARCHDIR/search.output
# Loop through your Search Criteria, placing info in our status file and writing match info
# to our output file.
while read SearchCriteria
do
echo $SearchCriteria >> $SEARCHDIR/search.status
grep "$SearchCriteria" $SearchFiles >> $SEARCHDIR/search.output 2>>$SEARCHDIR/search.error
done < SearchCriteria
rm $SEARCHDIR/$SearchFiles
exit
Sorry if I am stating the obvious, but a dot in regular expressions matches any character. To match a literal dot, use the regular expression [.] or backslash-escape the dot.
Yes, that is where I am stuck. I guess my real question should have been "How do you make a literal from an ip or url string?" I have tried the following, but both of the sed's give same results;
Code:
while read SearchCriteria
do
# sed "s/\./\\./g" $SearchCriteria
sed "s/\./[.]/g" $SearchCriteria
echo "$SearchCriteria" >> $SEARCHDIR/search.status
grep "$SearchCriteria" $SearchFiles >> $SEARCHDIR/search.output 2>>$SEARCHDIR/search.error
done < SearchCriteria
The simplest workaround would be to use fgrep instead of grep if your search criteria are always static strings. If you are lucky enough to have one which supports the -f option, you also don't need the loop at all.
If you want to neutralize any regex specials in the search string, try something like
Code:
grep `echo "$SearchCriteria" | sed -e 's/[][\\.*$^]/\\&/g'` $SearchFiles
Your attempt at using sed for this was not doing anything useful, I'm afraid. The above should hopefully work better, although it's completely off the top of my head (so I probably forgot a few of the regex specials) and different versions of sed use slightly different regex syntax (so yours probably has a slightly different set of special characters than mine).
The main misunderstanding was how to pass something to sed; it expects a file name (not a string to use as input) as an argument, or reads standard input; and simply prints any output, so to use the output in your script, you have to capture it with backquotes or something.
Just to top it off, here is a slightly more elegant and efficient way to code the loop:
Code:
sed -e 's/[][\\.*$^]/\\&/g' SearchCriteria |
while read regex; do
grep "$regex" $SearchFiles
done >>output 2>>error
Last edited by era; 06-17-2008 at 05:28 PM..
Reason: Note on fgrep -f patternfile; better sed loop
I thank you very much. I was spinning my wheels for a couple of days trying to figure out how to make the input records literal strings.
I would really like to be able to run in a loop for only one reason. The case that caused me to work on the scripts involved uncompressing 75 days worth of logs (3 -5 gb uncompressed) and searching for 36 ip ranges and 20 url strings.
When you start the script you never know where you are, or if the script is working (except with a ps -ef) until it completes and you view the output. I broke the original search files into one week ranges so the job would complete during my work hours. I was able to find the records I needed, but got a few that matched due to the "dot" wildcarding.
Do you think maybe the loop would work differently in a shell other than bash? I will test later. Now I need to get back to herding some cats too. Man, that is an extremely hard task!
Probably my sed script is not completely correct, in the general case or for your particular version of sed. Glad you sorted it out, anyway.
Oh, and you should probably use read -r when reading the regex from the sed script!
Also, try adding double quotes around the backticks.
Just out of curiosity, do you see what I see?
Code:
vnix$ echo 12.34.56.78 '^ick\\y.poo$' | sed -e 's/[][\\.*$^]/\\&/g'
12\.34\.56\.78 \^ick\\\\y\.poo\$
I don't think changing shells would change anything significantly, except if you switch to a shell which doesn't have support for the -r option to read. Maybe you have the line command instead; if so, try that.
For what it's worth, my cats are the cat(1) kind, not the felines (-:
Big cheezy grins from this side of the console. The -r was all that was needed to exclude special treatment of the \'s.
Now I can incorporate it into our log search tool and publish it for our firewall admin.
As for seeing what you see, yes, I do. So we evidently are using close, if not exact versions.
Thank you very much for your help. Folks like you, make folks like me, look good to the boss! Credits have already been given to you inside of the tool in our repository.
JB
Glad yours are not the feline versions... you would never get the time to help us out in the forums.
Hi
I'm having hard time here with below script. If i run script manually i see expected results but, if i keep this script in cron job i'm getting unexpected results. Unexpected results means even though condition is true,cronjob returning output of else condition.
This script and cronjob... (2 Replies)
Hi
I am comparing two files with comm -13 < (sort acc11.txt) < (sort acc12.txt) > output.txt
purpose: Get non matching records which are in acc12 but not in acc11...
TI am getting WRONG output.
Is there any constraints with record length with comm? The above files are the two consective ... (2 Replies)
I created 3 files with the identical data as follows
dial-peer voice 9999 pots
trunkgroup CO
list outgoing Local
translation-profile outgoing LOCAL-7-DIGITS-NO-PREPEND-97
preference 2
shutdown
destination-pattern 9......$
forward-digits 7
dial-peer voice 10000 pots
... (6 Replies)
I have been living with this problem with GNU sed v4.1.4 for a long time, but now I really need to figure it out.
When using a list in either an address or a search, the expression is matching lower and upper-case letters. works as it should.
For example, if I run
sed -nr "// p"... (7 Replies)
Hi,
I recently executed a find command that caused unexpected permission changes and we had to do a full system restore. Can someone please explain what this command would do?
find /staging/admin/scr * -exec chmod 755 '{}' +
It caused file permissions inside / to be modified strangely.
... (1 Reply)
Following script gives different results when triggered from Cron compared to when triggered from command line.
It is not able to assign values to some variables when triggered from cron.
Can any one help? Its a very simple script
Script -
#! /bin/ksh
sFile=$1
sEnv=$2
sWaitFile=$3... (1 Reply)
I must automatically monitor and manage a large number of boxes on our network.
I have been using perl/Net::Telnet and expect/telnet and also perl/ssh and expect/ssh to reach the command line of the remote boxes. Scripts are working but slow.
(Yes, I do use SNMP also but many boxes do not... (2 Replies)
Hello,
When I run this script, here's what I get:
Searching ...
found 1111
2222
3333
.....
7777
.....
8888
9999 in 95_test
Search completed.
I expected only to see what number was found in the file, not including the ones not found.
Thanks for your help!
#!/bin/sh (1 Reply)
i've reworked some code from an earlier post, and it isn't working as expected
i've simplified it to try and find the problem. i spent hours trying to figure out what is wrong, eventually thinking there was a bug in perl or a problem with my computer. but, i've tried it on 3 machines with the... (5 Replies)
06-19-2008
Unexpected Results (at least I did not expect them)