04-22-2018
In general doing this manually would be folly but I know at least that there are efforts at blocking all "bad" bots (bots with recognisable user agents)
One example is: Apache Ultimate Bad Bot Blocker (find on github).
It uses BrowserMatchNoCase or similar to match user agents and put them in a list (set an environment variable for it) which then, as a whole, is denied.
There's little point in blocking known URLs your server doesn't have, as opposed to the bots that try to access the ones you *do* have.
So blocking the URLs is pointless (ineffective if you don't have them, and disruptive if you do have them), you will have to block the activity itself.
Many bots (most bots) do not actually identify as a common user agent, even the hacker-type bots will just use something recognisable.
Alternatively, when a known URL gets hit that requires password authentication fail2ban is often employed to block individual IPs.
Last edited by dryden; 04-22-2018 at 08:38 AM..
Reason: Automatic merging not acceptable
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello, how to hide the full addres url, in apache web server. eg, www.example.org/www/pub/index.html, the address in browser only www.example.org .
Thank You. (2 Replies)
Discussion started by: blesets
2 Replies
2. UNIX for Advanced & Expert Users
Hi all,
How can I enable encoding of special characters present in URL?
eg
If the URL is
http://127.0.0.1/test.cgi?param1=test & test co
it shouldbe encoded to
http://127.0.0.1/test.cgi?param1=test%20%26%20test%20co
Thanks and Reagards,
uttam hoode (3 Replies)
Discussion started by: uttamhoode
3 Replies
3. Cybersecurity
Hello.
I have scenario where a Client send request to Server1.
Server1 send request to Server2.
Request are xmlHTTPRequest - need to get data (XML) from Server2 back to client.
Trying to use APACHE proxy...
Anyone can help?
What to download / configure / ...?
Thank you for your help. (1 Reply)
Discussion started by: ampo
1 Replies
4. Web Development
Hello.
I have scenario where a Client send request to Server1.
Server1 send request to Server2.
Request are xmlHTTPRequest - need to get data (XML) from Server2 back to client.
Trying to use APACHE proxy...
Anyone can help?
What to download / configure / ...?
Thank you for your... (2 Replies)
Discussion started by: ampo
2 Replies
5. Shell Programming and Scripting
i need help on redirecting apache conf file
i want redirect everything to www.example.com/home
example if i type a url
www.example.com/home/text1
i need that redirected to www.example.com/home (0 Replies)
Discussion started by: shehzad_m
0 Replies
6. Web Development
Hi All,
i'am facing a problem with urls that don't have a filestructure under DocumentRoot.
A URL like http://mydomain.com/applicationrew/Structure1/Structure2/some?parameter=key¶meter1=key1
Should be rewritet to something else.
Now i defined a Location like
<Location ~... (3 Replies)
Discussion started by: wuschelz
3 Replies
7. UNIX for Advanced & Expert Users
I need help in apache url redirection:
I have added the below command in httpd.conf and it is working fine.
Redirect http://xyz.com/site/homehttp://abc.com/site/home
Can we set a rule such that http://xyz.com/site/* -> http://abc.com/site/* is applied
For... (0 Replies)
Discussion started by: raghur77
0 Replies
8. Web Development
I need help in apache url redirection:
I have added the below command in httpd.conf and it is working fine.
Redirect http://xyz.com/site/homehttp://abc.com/site/home
Can we set a rule such that http://xyz.com/site/* -> http://abc.com/site/* is applied
For... (0 Replies)
Discussion started by: raghur77
0 Replies
9. Web Development
I am attempting to control redirections on my site using the Apache vhosts config.
I have two-letter regions setup, such as /fr, /de, /es, which I am currently ignoring as you will see from my vhosts file below. However I also have a 301 permanent redirect setup to redirect /cm (and /fr/cm,... (3 Replies)
Discussion started by: crmpicco
3 Replies
10. Red Hat
Hi folks,
I am trying to configure Apache webserver and also a virtual host inside this webserver.
For Global server config: /var/www/html/index.html
For virtual host config: /var/www/virtual/index.html
Both client10 & www10 are pointing to 192.168.122.10 IP address.
BUT, MY... (1 Reply)
Discussion started by: freebird8z
1 Replies
LEARN ABOUT SUSE
uri::url
URI::URL(3) User Contributed Perl Documentation URI::URL(3)
NAME
URI::URL - Uniform Resource Locators
SYNOPSIS
$u1 = URI::URL->new($str, $base);
$u2 = $u1->abs;
DESCRIPTION
This module is provided for backwards compatibility with modules that depend on the interface provided by the "URI::URL" class that used to
be distributed with the libwww-perl library.
The following differences exist compared to the "URI" class interface:
o The URI::URL module exports the url() function as an alternate constructor interface.
o The constructor takes an optional $base argument. The "URI::URL" class is a subclass of "URI::WithBase".
o The URI::URL->newlocal class method is the same as URI::file->new_abs.
o URI::URL::strict(1)
o $url->print_on method
o $url->crack method
o $url->full_path: same as ($uri->abs_path || "/")
o $url->netloc: same as $uri->authority
o $url->epath, $url->equery: same as $uri->path, $uri->query
o $url->path and $url->query pass unescaped strings.
o $url->path_components: same as $uri->path_segments (if you don't consider path segment parameters)
o $url->params and $url->eparams methods
o $url->base method. See URI::WithBase.
o $url->abs and $url->rel have an optional $base argument. See URI::WithBase.
o $url->frag: same as $uri->fragment
o $url->keywords: same as $uri->query_keywords
o $url->localpath and friends map to $uri->file.
o $url->address and $url->encoded822addr: same as $uri->to for mailto URI
o $url->groupart method for news URI
o $url->article: same as $uri->message
SEE ALSO
URI, URI::WithBase
COPYRIGHT
Copyright 1998-2000 Gisle Aas.
perl v5.12.1 2008-04-04 URI::URL(3)