MySQL Performance Problems


 
Thread Tools Search this Thread
The Lounge What is on Your Mind? MySQL Performance Problems
# 1  
Old 11-12-2019
MySQL Performance Problems

Just restarted MySQL a few times.

There seems to be a problem with MySQL performance because one table (our man page table) is too large and I need to move that table to a new database and out of the main forums DB.

That table is over 7 GB, bigger than the rest of the DB combined:

Code:
	neo_man_page_entry	347,938 rows 	MyISAM utf8_bin	size 7.1 GiB

This weekend I will move this table outside of the main forum DB and into it's only separate DB.

Hopefully, this will solve the performance issues; or at least lead me to a solution. I apologize for this MySQL performance problem; it's my fault for moving the man pages to the DB and preformatting the pages and keeping those preformatted pages in the DB as well; and hopefully moving this table outside of the main forum DB will help.

Thanks for your patience.
# 2  
Old 11-12-2019
The other possibility is that the search bots from China which are attacking the site are the problem:

Code:
Currently Active Users: 4410

This is the highest number we have seen in a few years, and is out of the ordinary; so I will block those bots with some quick PHP blocking code (below) and see if that makes a difference.
# 3  
Old 11-12-2019
Like so:

Code:
<?php
if (stripos($_SERVER['HTTP_USER_AGENT'], 'google') !== false) { // don't block googlebot
    define('NEO_SEND_SOME_BOTS_TO_403', false);
} else {
    define('NEO_SEND_SOME_BOTS_TO_403', true);
}

if (strlen($_SERVER['REMOTE_ADDR']) < 8) {
    $ip = '0.0.0.0';
} else {
    $ip = filter_var($_SERVER['REMOTE_ADDR'], FILTER_SANITIZE_STRING,FILTER_FLAG_STRIP_HIGH);
}

$checkbot = "SELECT bot_flag, country_iso2 FROM session WHERE country_iso2 IN ('RU','CN','KR','KP','UA','TW') AND userid = 0 AND host ='" . $ip . "' LIMIT 1";
$session_info = $vbulletin->db->query_first($checkbot);
if ($session_info['bot_flag'] == '1' && $session_info && NEO_SEND_SOME_BOTS_TO_403) {
    $file = '/var/log/apache2/debug/neo_global_country_bot_block_403.log';
    if ($neo_global['debug_block_with_403']) {
        error_log(date(DATE_RFC822) . ' SEND 403 Page: ' . $_SERVER['REMOTE_ADDR'] . ' Country: ' . $session_info['country_iso2'] . "\n", 3, $file);
    }
    header($_SERVER["SERVER_PROTOCOL"] . " 403 Forbidden", true, 403);
    include '/var/www/status/403.html';
    die;
}

# 4  
Old 11-12-2019
Which leads to:

Code:
linux:# tail -50 /var/log/apache2/debug/neo_global_country_bot_block_403.log
Tue, 12 Nov 19 08:58:32 -0600 SEND 403 Page: 182.111.240.115 Country: CN
Tue, 12 Nov 19 08:58:32 -0600 SEND 403 Page: 117.70.39.98 Country: CN
Tue, 12 Nov 19 08:58:35 -0600 SEND 403 Page: 117.60.39.21 Country: CN
Tue, 12 Nov 19 08:58:37 -0600 SEND 403 Page: 116.16.180.11 Country: CN
Tue, 12 Nov 19 08:58:39 -0600 SEND 403 Page: 91.122.30.68 Country: RU
Tue, 12 Nov 19 08:58:41 -0600 SEND 403 Page: 91.122.30.68 Country: RU
Tue, 12 Nov 19 08:58:41 -0600 SEND 403 Page: 223.244.152.107 Country: CN
Tue, 12 Nov 19 08:58:45 -0600 SEND 403 Page: 49.79.192.171 Country: CN
Tue, 12 Nov 19 08:58:46 -0600 SEND 403 Page: 114.104.184.190 Country: CN
Tue, 12 Nov 19 08:58:46 -0600 SEND 403 Page: 114.104.184.190 Country: CN
Tue, 12 Nov 19 08:58:48 -0600 SEND 403 Page: 117.69.144.83 Country: CN
Tue, 12 Nov 19 08:58:51 -0600 SEND 403 Page: 183.166.135.238 Country: CN
Tue, 12 Nov 19 08:58:52 -0600 SEND 403 Page: 123.179.131.100 Country: CN
Tue, 12 Nov 19 08:58:53 -0600 SEND 403 Page: 117.67.130.87 Country: CN
Tue, 12 Nov 19 08:58:56 -0600 SEND 403 Page: 114.237.28.37 Country: CN
Tue, 12 Nov 19 08:58:57 -0600 SEND 403 Page: 113.78.65.242 Country: CN
Tue, 12 Nov 19 08:58:58 -0600 SEND 403 Page: 119.7.152.128 Country: CN
Tue, 12 Nov 19 08:59:03 -0600 SEND 403 Page: 112.194.204.125 Country: CN
Tue, 12 Nov 19 08:59:07 -0600 SEND 403 Page: 49.83.242.142 Country: CN
Tue, 12 Nov 19 08:59:09 -0600 SEND 403 Page: 113.103.120.149 Country: CN
Tue, 12 Nov 19 08:59:10 -0600 SEND 403 Page: 114.223.161.134 Country: CN
Tue, 12 Nov 19 08:59:13 -0600 SEND 403 Page: 182.111.241.132 Country: CN
Tue, 12 Nov 19 08:59:16 -0600 SEND 403 Page: 101.75.156.131 Country: CN
Tue, 12 Nov 19 08:59:20 -0600 SEND 403 Page: 163.179.62.115 Country: CN
Tue, 12 Nov 19 08:59:21 -0600 SEND 403 Page: 117.69.241.84 Country: CN
Tue, 12 Nov 19 08:59:25 -0600 SEND 403 Page: 121.56.213.32 Country: CN
Tue, 12 Nov 19 08:59:25 -0600 SEND 403 Page: 49.89.84.54 Country: CN
Tue, 12 Nov 19 08:59:25 -0600 SEND 403 Page: 182.101.203.34 Country: CN
Tue, 12 Nov 19 08:59:26 -0600 SEND 403 Page: 91.122.30.68 Country: RU
Tue, 12 Nov 19 08:59:31 -0600 SEND 403 Page: 223.215.56.73 Country: CN
Tue, 12 Nov 19 08:59:32 -0600 SEND 403 Page: 36.23.217.84 Country: CN
Tue, 12 Nov 19 08:59:32 -0600 SEND 403 Page: 223.215.175.173 Country: CN
Tue, 12 Nov 19 08:59:35 -0600 SEND 403 Page: 180.125.17.185 Country: CN
Tue, 12 Nov 19 08:59:36 -0600 SEND 403 Page: 36.6.149.145 Country: CN
Tue, 12 Nov 19 08:59:45 -0600 SEND 403 Page: 37.9.113.198 Country: RU
Tue, 12 Nov 19 08:59:52 -0600 SEND 403 Page: 115.237.149.248 Country: CN
Tue, 12 Nov 19 08:59:52 -0600 SEND 403 Page: 117.43.174.94 Country: CN
Tue, 12 Nov 19 08:59:53 -0600 SEND 403 Page: 114.239.150.74 Country: CN
Tue, 12 Nov 19 08:59:54 -0600 SEND 403 Page: 36.4.85.49 Country: CN
Tue, 12 Nov 19 09:00:10 -0600 SEND 403 Page: 141.8.188.14 Country: RU
Tue, 12 Nov 19 09:00:10 -0600 SEND 403 Page: 59.33.107.241 Country: CN
Tue, 12 Nov 19 09:00:12 -0600 SEND 403 Page: 119.5.152.119 Country: CN
Tue, 12 Nov 19 09:00:13 -0600 SEND 403 Page: 141.8.142.50 Country: RU
Tue, 12 Nov 19 09:00:15 -0600 SEND 403 Page: 223.215.56.88 Country: CN
Tue, 12 Nov 19 09:00:20 -0600 SEND 403 Page: 114.98.173.136 Country: CN
Tue, 12 Nov 19 09:00:27 -0600 SEND 403 Page: 180.95.169.52 Country: CN
Tue, 12 Nov 19 09:00:31 -0600 SEND 403 Page: 117.57.62.8 Country: CN
Tue, 12 Nov 19 09:00:41 -0600 SEND 403 Page: 49.87.135.33 Country: CN
Tue, 12 Nov 19 09:00:50 -0600 SEND 403 Page: 123.130.129.40 Country: CN
Tue, 12 Nov 19 09:00:50 -0600 SEND 403 Page: 60.168.86.163 Country: CN

# 5  
Old 11-13-2019
Added some quick code to turn on and off various "bot blocking code" based on the load average of the site.

Very simple example:

Code:
<?php
$quickload = getLoad();
if ($quickload > 6) {
    $checkbot = "SELECT bot_flag, country_iso2 FROM session WHERE country_iso2 IN ('RU','CN','KR','KP','UA','TW') AND userid = 0 AND host ='" . $ipFilter . "' LIMIT 1";

    $session_info = $vbulletin->db->query_first($checkbot);
    if ($session_info['bot_flag'] == '1' && $session_info && NEO_SEND_SOME_BOTS_TO_403) {
        $file = '/var/log/apache2/debug/neo_global_country_bot_block_403.log';
        if ($neo_global['debug_block_with_403']) {
            error_log(date(DATE_RFC822) . ' SEND 403 Page: ' . $_SERVER['REMOTE_ADDR'] . ' Country: ' . $session_info['country_iso2'] ." Load: ".$quickload. "\n", 3, $file);
        }
        header($_SERVER["SERVER_PROTOCOL"] . " 403 Forbidden", true, 403);
        include '/var/www/status/403.html';
        die;
    }
}

function getLoad()
{
  $string=file_get_contents('/proc/loadavg');
  $loadavg=explode(' ', $string);
  if($loadavg[0]){
        return $loadavg[0];
  }else{
        return 0;
  }       
}

# 6  
Old 11-13-2019
Update:

Have made a few changes to help with the DB load during peak times.
  • Coped the huge 7 GB man page table over to another DB.
  • Moved three busy man page DB queries over to the new database and off the main forum DB.
  • On the last man page table still in use in the main DB, added some code to stop man pages from being served from the DB when the load is very high (serve from a unix command line function versus the DB).

Let's see today if things improve or not, today.

TODO:
  1. Test current code and config.
  2. Move the main man page DB query off the main forum DB and to the new DB.
  3. Delete the man page table from the main DB (when all is tested and done).
# 7  
Old 11-14-2019
Update:

Have completely moved all DB queries regarding man pages to a new database (outside of the main site DB).

After testing, will delete the table of man pages from the main DB.

Deleting this table from the main DB with reduce the size of the main DB by 75%. This, in turn, reduces the sizes of the daily backups and data dumps by 75%, etc. etc.

Note: Restarted MySQL a few times today.....
Login or Register to Ask a Question

Previous Thread | Next Thread

4 More Discussions You Might Find Interesting

1. Programming

Xlib - Rotation and interpolation of pixmap - Performance problems

I need to rotate a pixmap in XLib with some kind of interpolation to reduce the aliasing. I came up with the following code, which uses bilinear interpolation. It works fine: the rotated image looks perfect, but unfortunately it takes 5 or 6 seconds for each rotation. (in a 300x300, 16 colours... (5 Replies)
Discussion started by: mghis
5 Replies

2. AIX

AIX 5.3 performance problems

Hello, I encounter some performance issues on my AIX 5.3 server running in a LPAR on a P520. How do I investigate performance issues in AIX. Is there any kind of procedure that takes me to the steps to investigate my server and find the sub systems that is causing the issues? The performance... (1 Reply)
Discussion started by: petervg
1 Replies

3. SCO

CPU Performance Problems on VMWARE

hi We have migrated SCO 5.0.6 into ESX4, but the VM eats 100% of the virtual CPU. Here is top print from the SCO VM: last pid: 16773; load averages: 1.68, 1.25, 0.98 02:08:41 79 processes: 75 sleeping, 2 running, 1 zombie, 1 onproc CPU states: 0.0% idle, 17.0% user,... (7 Replies)
Discussion started by: ccc
7 Replies

4. AIX

Performance and paging problems

... a disk drive to be 100% busy? hdisk0 100.0 1.3K 342.7 1.3K 22.0 PgspIn 651 % Noncomp 75.5 hdisk1 100.0 1.3K 320.2 1.2K 20.0 PgspOut 6 % Client 75.5 It's really slowing down performance on my system and I would like to know what is causing this. ... (2 Replies)
Discussion started by: bbbngowc
2 Replies
Login or Register to Ask a Question