04-22-2018
In general doing this manually would be folly but I know at least that there are efforts at blocking all "bad" bots (bots with recognisable user agents)
One example is: Apache Ultimate Bad Bot Blocker (find on github).
It uses BrowserMatchNoCase or similar to match user agents and put them in a list (set an environment variable for it) which then, as a whole, is denied.
There's little point in blocking known URLs your server doesn't have, as opposed to the bots that try to access the ones you *do* have.
So blocking the URLs is pointless (ineffective if you don't have them, and disruptive if you do have them), you will have to block the activity itself.
Many bots (most bots) do not actually identify as a common user agent, even the hacker-type bots will just use something recognisable.
Alternatively, when a known URL gets hit that requires password authentication fail2ban is often employed to block individual IPs.
Last edited by dryden; 04-22-2018 at 08:38 AM..
Reason: Automatic merging not acceptable
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello, how to hide the full addres url, in apache web server. eg, www.example.org/www/pub/index.html, the address in browser only www.example.org .
Thank You. (2 Replies)
Discussion started by: blesets
2 Replies
2. UNIX for Advanced & Expert Users
Hi all,
How can I enable encoding of special characters present in URL?
eg
If the URL is
http://127.0.0.1/test.cgi?param1=test & test co
it shouldbe encoded to
http://127.0.0.1/test.cgi?param1=test%20%26%20test%20co
Thanks and Reagards,
uttam hoode (3 Replies)
Discussion started by: uttamhoode
3 Replies
3. Cybersecurity
Hello.
I have scenario where a Client send request to Server1.
Server1 send request to Server2.
Request are xmlHTTPRequest - need to get data (XML) from Server2 back to client.
Trying to use APACHE proxy...
Anyone can help?
What to download / configure / ...?
Thank you for your help. (1 Reply)
Discussion started by: ampo
1 Replies
4. Web Development
Hello.
I have scenario where a Client send request to Server1.
Server1 send request to Server2.
Request are xmlHTTPRequest - need to get data (XML) from Server2 back to client.
Trying to use APACHE proxy...
Anyone can help?
What to download / configure / ...?
Thank you for your... (2 Replies)
Discussion started by: ampo
2 Replies
5. Shell Programming and Scripting
i need help on redirecting apache conf file
i want redirect everything to www.example.com/home
example if i type a url
www.example.com/home/text1
i need that redirected to www.example.com/home (0 Replies)
Discussion started by: shehzad_m
0 Replies
6. Web Development
Hi All,
i'am facing a problem with urls that don't have a filestructure under DocumentRoot.
A URL like http://mydomain.com/applicationrew/Structure1/Structure2/some?parameter=key¶meter1=key1
Should be rewritet to something else.
Now i defined a Location like
<Location ~... (3 Replies)
Discussion started by: wuschelz
3 Replies
7. UNIX for Advanced & Expert Users
I need help in apache url redirection:
I have added the below command in httpd.conf and it is working fine.
Redirect http://xyz.com/site/homehttp://abc.com/site/home
Can we set a rule such that http://xyz.com/site/* -> http://abc.com/site/* is applied
For... (0 Replies)
Discussion started by: raghur77
0 Replies
8. Web Development
I need help in apache url redirection:
I have added the below command in httpd.conf and it is working fine.
Redirect http://xyz.com/site/homehttp://abc.com/site/home
Can we set a rule such that http://xyz.com/site/* -> http://abc.com/site/* is applied
For... (0 Replies)
Discussion started by: raghur77
0 Replies
9. Web Development
I am attempting to control redirections on my site using the Apache vhosts config.
I have two-letter regions setup, such as /fr, /de, /es, which I am currently ignoring as you will see from my vhosts file below. However I also have a 301 permanent redirect setup to redirect /cm (and /fr/cm,... (3 Replies)
Discussion started by: crmpicco
3 Replies
10. Red Hat
Hi folks,
I am trying to configure Apache webserver and also a virtual host inside this webserver.
For Global server config: /var/www/html/index.html
For virtual host config: /var/www/virtual/index.html
Both client10 & www10 are pointing to 192.168.122.10 IP address.
BUT, MY... (1 Reply)
Discussion started by: freebird8z
1 Replies
LEARN ABOUT SUSE
html::linkextor
HTML::LinkExtor(3) User Contributed Perl Documentation HTML::LinkExtor(3)
NAME
HTML::LinkExtor - Extract links from an HTML document
SYNOPSIS
require HTML::LinkExtor;
$p = HTML::LinkExtor->new(&cb, "http://www.perl.org/");
sub cb {
my($tag, %links) = @_;
print "$tag @{[%links]}
";
}
$p->parse_file("index.html");
DESCRIPTION
HTML::LinkExtor is an HTML parser that extracts links from an HTML document. The HTML::LinkExtor is a subclass of HTML::Parser. This means
that the document should be given to the parser by calling the $p->parse() or $p->parse_file() methods.
$p = HTML::LinkExtor->new
$p = HTML::LinkExtor->new( $callback )
$p = HTML::LinkExtor->new( $callback, $base )
The constructor takes two optional arguments. The first is a reference to a callback routine. It will be called as links are found. If
a callback is not provided, then links are just accumulated internally and can be retrieved by calling the $p->links() method.
The $base argument is an optional base URL used to absolutize all URLs found. You need to have the URI module installed if you provide
$base.
The callback is called with the lowercase tag name as first argument, and then all link attributes as separate key/value pairs. All
non-link attributes are removed.
$p->links
Returns a list of all links found in the document. The returned values will be anonymous arrays with the following elements:
[$tag, $attr => $url1, $attr2 => $url2,...]
The $p->links method will also truncate the internal link list. This means that if the method is called twice without any parsing
between them the second call will return an empty list.
Also note that $p->links will always be empty if a callback routine was provided when the HTML::LinkExtor was created.
EXAMPLE
This is an example showing how you can extract links from a document received using LWP:
use LWP::UserAgent;
use HTML::LinkExtor;
use URI::URL;
$url = "http://www.perl.org/"; # for instance
$ua = LWP::UserAgent->new;
# Set up a callback that collect image links
my @imgs = ();
sub callback {
my($tag, %attr) = @_;
return if $tag ne 'img'; # we only look closer at <img ...>
push(@imgs, values %attr);
}
# Make the parser. Unfortunately, we don't know the base yet
# (it might be different from $url)
$p = HTML::LinkExtor->new(&callback);
# Request document and parse it as it arrives
$res = $ua->request(HTTP::Request->new(GET => $url),
sub {$p->parse($_[0])});
# Expand all image URLs to absolute ones
my $base = $res->base;
@imgs = map { $_ = url($_, $base)->abs; } @imgs;
# Print them out
print join("
", @imgs), "
";
SEE ALSO
HTML::Parser, HTML::Tagset, LWP, URI::URL
COPYRIGHT
Copyright 1996-2001 Gisle Aas.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.12.1 2009-02-09 HTML::LinkExtor(3)