Shell Programming and Scripting

Hello all,

I am in need of assistance in creating a script that will remove a specified block of text from multiple .htaccess files. (roughly 1000 files)

I am attempting to help with a project to clean up a linux server that has a series of unwanted url rewrites in place, as well as some rules that need to stay in place.

The series of rewrite rules that needs to be removed is the same in each file, and the htaccess files are spread throughout 100's of directories within the server.

I am not all that great with command line scripts but I know enough to navigate the server via shell and implement fairly simple scripts. Unfortunately no on else within the company has any experience within maintaining a linux server, so I was delegated to take the lead on this project.

I believe this task can be accomplished using a combination of find, grep, and sed (or awk) but I could use some help in combining these functions to complete the task.

I was able to output a list of all the files that contain the matching rules to a text file to see how extensive the changes that need to be made are going to be using the following:

find /var/www/html/vhosts -name ".htaccess" -exec grep -l "Text To Match" > output.txt {} \;

This seemed to spit out the requested information in a timely matter, now I need to add sed into the equation to actually modify the files that I found, using either the text file containing the path's to each of these files, or piping the output into sed instead of to the text file.

This is where it gets a little bit complicated and the confusion for me sets in. I figured rather than spend countless hours searching google, I would seek the assistance of some others.. that hopefully are a little more knowledgeable than me at the command line.

Can anyone chime in and explain to me how i can incorporate sed or awk into the equation to modify all of these htaccess files to only remove the block of text I specify, while leaving the remaining text in place.

Any help or direction would be greatly appreciated.

Thanks,
BoxX

Welcome to the forum. Please use code tags when posting any code or output.

To do the deletion, you could do the following:

Code:

sed '/beg/,/end/ d' filename

which will delete all lines between the occurence of 'beg' and occurence of 'end' and print the result on stdout. 'beg' and 'end' can be regular expressions. sed's pattern matching is greedy.
You can delete it from the file (after you tested your command) using sed's -i (in-place editing) switch.
You can use find(1)'s -exec command to pass the filenames to sed, just like you were doing with grep:

Code:

find /var/www/html/vhosts -name ".htaccess" -exec sed -i '/BegRE/,/endRE/ d' {} \;

If you can be more specific with what does your 'block of text' to delete look like, we can help you build the regex to match (the start and end point) as closely as possible.
You could also specify line numbers, if that is constant across the files (doubtful).

Thanks for the response,

That looks to be exactly what I'm looking for..

and example of the text I'm referring to that needs to be removed is as follows:

Code:

# THISMATCHESINEVERYENTRY
RewriteEngine On
RewriteCond %{REQUEST_METHOD}   ^GET$
RewriteCond %{HTTP_REFERER}     ^(http\:\/\/)?([^\/\?]*\.)?(google\.|yahoo\.|bing\.|msn\.|yandex\.|ask\.|excite\.|altavista\.|netscape\.|aol\.|hotbot\.|goto\.|infoseek\.|mamma\.|alltheweb\.|lycos\.|search\.|metacrawler\.|rambler\.|mail\.|dogpile\.|ya\.|\/search\?).*$   [NC]
RewriteCond %{HTTP_REFERER}     !^.*(q\=cache\:).*$   [NC]
RewriteCond %{HTTP_USER_AGENT}  !^.*(bing|Accoona|Ace\sExplorer|Amfibi|Amiga\sOS|apache|appie|AppleSyndication).*$   [NC]
# THISMATCHESINEVERYENTRY

The start and end points are clearly marked in each .htaccess file, so deleting everything from the opening match to the closing match is desired.

There are no special characters in the start or end tags, except # and spaces, but there are alot of special characters in the contained block.

would...

Code:

find /var/www/html/vhosts -name ".htaccess" -exec sed -i '/# THISMATCHESINEVERYENTRY/,/# THISMATCHESINEVERYENTRY/ d' {} \;

suffice?

---------- Post updated at 02:19 AM ---------- Previous update was at 12:50 AM ----------

Code:

find /var/www/html/vhosts -name ".htaccess" -exec sed -i '/# THISMATCHESINEVERYENTRY/,/# THISMATCHESINEVERYENTRY/ d' {} \;

seems to be the winner

Thanks for the help. breaking it down, that makes perfect sense. And seems to work in the few test runs I've made.

Kudos ^^

-BoxX

It's always good to make the regex as specific as possible. Your command will work, but it would also match a line:

Code:

# this is what we use to separate block: # THISMATCHESINEVERYENTRY

I'd use anchors:

Code:

sed '/^# THISMATCHESINEVERYENTRY *$/,/^# THISMATCHESINEVERYENTRY *$/ d'

would only match a line starting with '# THISMATCHESINEVERYENTRY' and zero or more spaces until end of line.

Metacharacters in the lines being deleted don't matter at all.

I made a mistake about sed's matching being greedy. Although it's true for pattern matching, it's not true for matching ranges of lines. When specifying ranges of lines, sed turns a flag on or off, when it encounters the pattern.

Depending on what you want to filter out, take care about this :
if you use # THISMATCHESINEVERYENTRY as the pattern triggering the start/stop deletion, you should make sure that this pattern appear an even number of time.

Consider the following example and see what append with a odd number of that pattern :

Code:

$ cat tst
blabla
# THISMATCHESINEVERYENTRY
filtered out
filtered out
# THISMATCHESINEVERYENTRY
blabla
# THISMATCHESINEVERYENTRY
filtered out
filtered out
filtered out
tutu
# THISMATCHESINEVERYENTRY
toto
# THISMATCHESINEVERYENTRY
filtered out
titi
filtered out

Code:

$ sed '/# THISMATCHESINEVERYENTRY/,/# THISMATCHESINEVERYENTRY/ d' tst
blabla
blabla
toto
$

As you see, the end of the file will be filtered out.

---------- Post updated at 11:37 AM ---------- Previous update was at 11:33 AM ----------

If you choose the same /startRE/ and /endRE/ pattern, every odd occurence of the pattern will start the deletion, make sure that the file you want to parse fit the format you need (so you will avoid bad surprise of unexpected deletion)