Trying to do a compare with multiple lines in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Trying to do a compare with multiple lines in a file
# 1  
Old 02-23-2010
Trying to do a compare with multiple lines in a file

Hey guys I am having a problem with being able to find unused profiles in a configuration check script I am trying to create for accountability purposes for managing a large number of systems. What I am trying to do is run a script that will look at the raw config data in a file and pull all the profiles entries that aren't being used. The file that I am pulling has a ton of information but what I am trying to do is pull the various profile data and compare it to the virtual servers that I have configured and report the profiles that are not being used at this time.Here is the output of Config Data from that script:

Code:
profile client test1 {
   defaults from clientssl
   key "blank_ssl.key"
   cert "blank_ssl.crt"
   chain "WF-CA-BUNDLE.crt"
   ca file "WF-CA-BUNDLE.crt"
   client cert ca "WF-CA-BUNDLE.crt"
   ciphers "DEFAULT:+DES-CBC-SHA"
   peer cert mode require
   authenticate depth 3
}
profile client test2 {
   defaults from clientssl
   key "blank_ssl.key"
   cert "blank_ssl.crt"
   chain "WF-CA-BUNDLE.crt"
   ca file "WF-CA-BUNDLE.crt"
   client cert ca "WF-CA-BUNDLE.crt"
   ciphers "DEFAULT:+DES-CBC-SHA"
   peer cert mode require
   authenticate depth 3
}
profile client test3 {
   defaults from clientssl
   key "blank_ssl.key"
   cert "blank_ssl.crt"
   chain "WF-CA-BUNDLE.crt"
   ca file "WF-CA-BUNDLE.crt"
   client cert ca "WF-CA-BUNDLE.crt"
   ciphers "DEFAULT:+DES-CBC-SHA"
   peer cert mode require
   authenticate depth 3
}
virtual test_1 {
   limit 1
   destination 1.2.3.4:telnet
   ip protocol tcp
   rules test1_rule
   profiles test1
}
virtual test_2 {
   limit 1
   destination 1.2.3.4:telnet
   ip protocol tcp
   rules test1_rule
   profiles 
	test2
	tcp
}
virtual test_3 {
   limit 1
   destination 1.2.3.4:telnet
   ip protocol tcp
   rules test1_rule
   profiles 
	test2
}

So what I am trying to do is pull all the above profile entries and compare it to the entries in each of the Virtuals so I want it retun that test3 isn't being used. As you can see it can either be on the same line as "profiles ...." or could be the following lines before the "}'

I am using the following method but doesn't work on the lines that follow the "profiles"

Code:
echo "-------------------------------------------------------------------------"
echo "|"
echo "| Checking for orphaned monitors..."
echo "|"
profiles=`grep -i "^profile" ${CONF} | cut -f3 -d" "`
for i in $profiles; do
   profilecheck=`grep "profiles $i" ${CONF}`
   [ "${profilecheck}" != "" ] || echo "|   profile $i may be unused."
done
echo "|"
echo "-------------------------------------------------------------------------"

Any assistance would greatly be appreciated.
# 2  
Old 02-23-2010
Hi.

Here is a quick hack that produces the desired output using standard commands. Many intermediate files are produced so that you can see the steps of refinement. One nonstandard command is used, cgrep, complex grep:
cgrep home page
which has compiled easily for me in many environments.

Basically, the data file is parsed into two sections, each of which is refined, until the profiles names of each category is placed into separate files. Then either comm or grep may be used.
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate parsing, sorting, filtering.

# Infrastructure details, environment, commands for forum posts. 
set +o nounset
LC_ALL=C ; LANG=C ; export LC_ALL LANG
echo ; echo "Environment: LC_ALL = $LC_ALL, LANG = $LANG"
echo "(Versions displayed with local utility \"version\")"
c=$( ps | grep $$ | awk '{print $NF}' )
version >/dev/null 2>&1 && s=$(_eat $0 $1) || s=""
[ "$c" = "$s" ] && p="$s" || p="$c"
version >/dev/null 2>&1 && version "=o" $p grep cgrep awk sed tr sort comm
set -o nounset
echo

FILE=${1-data1}

# Remove debris.
rm -f t[1-9]

echo " Sample of $(wc -l <$FILE) lines in data file $FILE:"
head -5 $FILE
echo -----
tail -5 $FILE

echo
echo " Results:"
grep -w profile $FILE |
tee t1 |
cat > defs

cgrep -D +w '^}' virtual $FILE |
tee t2 |
cat > virs

awk '{ print $3 }' defs |
tee t3 |
cat > defs-names

cgrep -D +I2 +w '^}' profile virs |
tee t4 |
sed '/^[ \t]*$/d' |
sort > virs-names

sed 's/profiles//' virs-names |
tee t5 |
tr '\n' ' ' |
tee t6 |
sed -e 's/^ *//' -e 's/[ \t][ \t]*/ /g' |
tee t7 |
tr ' ' '\n' |
tee t8 |
sed '/^[ \t]*$/d' |
sort -u > virs-final

# use:
# comm -13 virs-final defs-names
# if unmatched profiles should not be displayed.

comm -3 virs-final defs-names

exit 0

producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
GNU grep 2.5.3
cgrep - (local: ~/executable/cgrep May 29 2009 )
GNU Awk 3.1.5
GNU sed version 4.1.5
tr (GNU coreutils) 6.10
sort (GNU coreutils) 6.10
comm (GNU coreutils) 6.10

 Sample of 58 lines in data file data1:
profile client test1 {
   defaults from clientssl
   key "blank_ssl.key"
   cert "blank_ssl.crt"
   chain "WF-CA-BUNDLE.crt"
-----
   rules test1_rule
   profiles 
	test2
}


 Results:
tcp
	test3

There clearly are other solutions, and someone will probably drop by with one soon.

Best wishes ... cheers, drl
# 3  
Old 02-23-2010
You could try if this works out...
Code:
awk '$1=="profile"{A[$3]=1} $1=="profiles"{sub($1,"");p=1} $1=="}"{p=0} p{for(i=1;i<=NF;i++)A[$i]--}
     END{for(i in A)if (A[i]>0) print i}' infile

Code:
test3

# 4  
Old 02-24-2010
Thx Guys. I was able to tweak your suggestions to use. I appreciate your assistance.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

2. Shell Programming and Scripting

How to compare 2 files and create a result file with unmatched lines from first file.?

HI, I have 2 text files. file1 and file2. file1.txt (There are no duplicates in this file) 1234 3232 4343 3435 6564 6767 1213 file2.txt 1234,wq,wewe,qwqw 1234,as,dfdf,dfdf 4343,asas,sdds,dsds 6767,asas,fdfd,fdffd I need to search each number in file1.txt in file2.txt's 1st... (6 Replies)
Discussion started by: Little
6 Replies

3. Shell Programming and Scripting

Awk match multiple columns in multiple lines in single file

Hi, Input 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr21.fa chr3.fa 7488 7389 chr1.fa chr1.fa 3546 9887 chr9.fa chr5.fa 7898 7387 chrX.fa chr3.fa Desired Output 7488 7389 chr1.fa chr1.fa 2 3546 9887 chr5.fa chr9.fa 2... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

4. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

5. Shell Programming and Scripting

compare lines in a file

Hi Folks, I need to compare the cron's timings from a text file. Need to display how much time does it took for that job. For example i have the below txt file, I have cron1 started at 05:23:15 and completed at 05:25:57, now i need to find how much time did it took to complete corn1 job for... (8 Replies)
Discussion started by: Sendhil.Kumaran
8 Replies

6. UNIX for Dummies Questions & Answers

grep command to find multiple strings in multiple lines in a file.

I want to search files (basically .cc files) in /xx folder and subfolders. Those files (*.cc files) must contain #include "header.h" AND x() function. I am writing it another way to make it clear, I wanna list of *.cc files that have 'header.h' & 'x()'. They must have two strings, header.h... (2 Replies)
Discussion started by: ritikaSharma
2 Replies

7. Shell Programming and Scripting

Compare lines within a file

could someone write a short script which is able to take a text file input and compare lines 1 and 2, 3 and 4, 5 and 6 ... ... until the end of the file. And to prompt if the lines are not equal. Thank you! (10 Replies)
Discussion started by: c_lady
10 Replies

8. Shell Programming and Scripting

How do I compare the last two lines of a file?

Hi, I want to compare the last two lines of a files, specifically characters 32 - 50 in both lines and generate an exit code if the range of characters do not match. Please advise. Thanks in advance! (2 Replies)
Discussion started by: limshady411
2 Replies

9. Shell Programming and Scripting

Help! How to compare two lines in a file

Hello, I am newcomer and sorry for this simple question. I want to how to compare two lines in a file? For example, to compare the first line and the second line of a file to know if they are same? Thanks in advance! Leon (3 Replies)
Discussion started by: sabertooth2000
3 Replies

10. Shell Programming and Scripting

retrieved multiple lines on multiple places in a file

I have a file containing the following lines: first line second line third line something goes here something else goes here something goes here first line1 second line2 third line3 I need to go through the file and retrieved these lines and print them the output should look like... (8 Replies)
Discussion started by: dala
8 Replies
Login or Register to Ask a Question