CentOS 6 ran out of space, need to reclaim it


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers CentOS 6 ran out of space, need to reclaim it
# 15  
Old 08-27-2019
Example:

Code:
myserver:/etc/mysql# cat mysql.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

Code:
myserver:/etc/mysql# cd mysql.conf.d/
myserver:/etc/mysql/mysql.conf.d# l
total 36
-rw-r--r-- 1 root root 4991 Jul 19 20:24 mysqld.cnf
-rw-r--r-- 1 root root 3453 Jun 14 23:32 mysqld.cnf.back
-rw-r--r-- 1 root root 4555 Jun 21 23:42 mysqld.cnf.back2

Code:
myserver:/etc/mysql/mysql.conf.d# cat mysqld.conf



Code:
myserver:/etc/mysql/mysql.conf.d# cat mysqld.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
log_error_verbosity = 3
ft_min_word_len=3
open_files_limit = 65535
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address		= 127.0.0.1
#
# * Fine Tuning
#


# OMITTED FROM CUT AND PASTE EXAMPLE

You really care about the entry in red above.

Best you post your entire mysql.cnf file for us to review. Then, I can better tell you exactly what is going on and how to fix your problem Smilie
# 16  
Old 08-27-2019
In parallel let me help you to look for the files that fill your / file system.
Your post#11 explains some things. (In your post#1 you said that "df -h" gave output for "/" only. That was misleading.)
Can you shortly umount the /mnt/backup and look for files? Such files are invisible when mounted.
Code:
umount /mnt/backup
ls -lah /mnt/backup
du -sh /mnt/backup
mount /mnt/backup

(Just seeing that Peasant has the same idea in post#13.)
These 2 Users Gave Thanks to MadeInGermany For This Post:
# 17  
Old 08-27-2019
Following up on my request to post your mysql.cnf file.... take a look at this example:

Code:
just_a_mysql_server:/var/lib/mysql# ls -l
total 254144
-rw-r----- 1 mysql mysql       56 Sep  5  2018 auto.cnf
-rw-r----- 1 mysql mysql     4134 Jul 19 20:24 ib_buffer_pool
-rw-r----- 1 mysql mysql 79691776 Aug 27 04:26 ibdata1
-rw-r----- 1 mysql mysql 50331648 Aug 27 04:26 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Jun 18 03:07 ib_logfile1
-rw-r----- 1 mysql mysql 79691776 Aug 27 04:29 ibtmp1

Note the HUGE binary log files above:


Quote:
The file ibdata1 is the system tablespace for the InnoDB infrastructure.

It contains several classes for information vital for InnoDB

....


But before we can fully analyze and create a solution for you (which I think is very easy to do, BTW), we need to see your mysql.cnf file Smilie
# 18  
Old 08-27-2019
Thank you for taking your time helping me.

here is my.cnf, it is shorter than your example, but I guess the initial set up was to leave a lot as a default:

Code:
less /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
port=3306
bind-address=0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

/var/lib/mysql didn't have any huge files:

Code:
ls -lh
total 29M
-rw-rw---- 1 mysql mysql  18M Aug 21 03:15 ibdata1
-rw-rw---- 1 mysql mysql 5.0M Aug 27 07:21 ib_logfile0
-rw-rw---- 1 mysql mysql 5.0M Aug 27 07:21 ib_logfile1
drwx------ 2 mysql mysql 4.0K Oct 31  2014 mysql
srwxrwxrwx 1 mysql mysql    0 Aug 27 07:21 mysql.sock
drwx------ 2 mysql mysql  20K Oct 31  2014 SUGARBAK
drwx------ 2 mysql mysql  20K Jul 24  2018 sugarcrm




Code:
tail /var/log/mysqld.log
190827  7:35:26  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
190827  7:35:26  InnoDB: Started; log sequence number 0 8747645
190827  7:35:26 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190827  7:35:26 [ERROR] Can't start server: can't create PID file: No space left on device
190827 07:35:26 mysqld_safe Number of processes running now: 0
190827 07:35:26 mysqld_safe mysqld restarted

# 19  
Old 08-27-2019
Do you have a directory as follows:

Code:
/etc/mysql

?

I am guessing.... no ... correct?
# 20  
Old 08-27-2019
i suggest you create a directory:

Code:
/home/var/lib/mysql

and copy your entire /var/lib/mysql dir to the new directory and then make sure your mysql process can write to the new dir.

Then edit your my.cnf file and change the data dir to your new dir in /home (for now just for testing) and start mysqld.

If you do this correctly mysqld should start OK.
This User Gave Thanks to Neo For This Post:
# 21  
Old 08-27-2019
Quote:
Originally Posted by Peasant

I was about to be walked through extending ext4 partition as a suggestion from one of the forums - which would've worked too (if done correctly), but ....
Yeah, I would not have done the "extend the root partition" approach. It would work but it is suboptimal.

It would be better to create a new partition for the mysql data and move it off the root partition. I try to keep as much user data, dumps and backups off the root partition as I can, generally speaking.


You might still consider doing this, as you are still have your main apps data directory on your root partition.

My suggestion to you, after you got mysqld running again on the (temp) /home partition, was to ask you to create a new partition for your db data. Normally I call this, not surprisingly:

Code:
/data

Anyway, you are well on your way now. Well done.
This User Gave Thanks to DannyBoyCentOS For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Safe way to shrink lvm vg_*-lv_swap partition and reclaim freed space on Linux?

Hello, # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 38.2G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 ... (2 Replies)
Discussion started by: centosadmin
2 Replies

2. AIX

How to reclaim hard disks and IP's in AIX?

Hello I recently received a request to reclaim hard disks and IP addresses within an AIX system(s). THe file systems are no longer in use and the client has indicated that it is OK to remove them and reclaim the disks and release the IP's. Now, since the file systems belong to a Volume group I... (8 Replies)
Discussion started by: Joseph Sabo
8 Replies

3. Red Hat

How to Upgrade Centos 5.7 using Centos 5.8 ISO image on Vmware workstation

Dear Linux Experts, On my windows 7 desktop with the help of Vmware workstation (Version 7.1), created virtual machine and installed Centos 5.7 successfully using ISO image. Query : Is this possible to upgrade the Centos 5.7 using Centos 5.8 ISO image to Centos version 5.8?.. if yes kindly... (2 Replies)
Discussion started by: Ananthcn
2 Replies

4. Red Hat

On CentOS, moving space from large free directory to another

Hi. My "/usr" folder is running out of space. My "/home" folder is quite large and has a lot of free space. As follows: Filesystem Type Size Used Avail Use% Mounted on ... /dev/sda5 ext3 9.7G 2.6G 6.7G 28% / /dev/sda7 ext3 152G 16G 128G 11% /home /dev/sda3 ... (7 Replies)
Discussion started by: pkiula
7 Replies

5. Linux

How to reclaim the space which i used to increse the swap space on Xen,

Hi, i have done a blunder here, i increased the swap space on Xen5.6 server machine using below steps :- 1056 dd if=/dev/zero of=/root/myswapfile bs=1M count=1024 1057 ls -l /root/myswapfile 1058 chmod 600 /root/myswapfile 1059 mkswap /root/myswapfile 1060 swapon /root/myswapfile ... (1 Reply)
Discussion started by: apm
1 Replies

6. Shell Programming and Scripting

how to find whether a script ran or not

Hi, I have written a script and placed in an application and the script can be executed manually only. But somehow one of the method in the script is being called and bringing the application down. But we are not able to find any instance of script running. Is there a way to findout whether the... (1 Reply)
Discussion started by: Satyak
1 Replies

7. Shell Programming and Scripting

Identify if ran by su or sudo?

Recently I was on an operational call and heard the people running my code placing the code in the /tmp directory and running as root. I had not planned on that. So I want to add some checks to my code (using ksh93): # ---------- ---------- ---------- # root not allowed to run this #... (3 Replies)
Discussion started by: ericdp63
3 Replies

8. UNIX for Dummies Questions & Answers

Reclaim deleted disk space

I have a disk space issue on one of my unix servers. it is showing 98% full.. i found the offending folder and removed it. but i have not reclaimed the disk space. is there another command that i need ? thank you in advance for any assistance. (4 Replies)
Discussion started by: JanSP
4 Replies

9. Shell Programming and Scripting

which user ran which command

can we come to know all the command ran bya user for last 1 day (1 Reply)
Discussion started by: narang.mohit
1 Replies

10. Filesystems, Disks and Memory

Ran out of space on /dev/root partition

hi, I have a SCO unix server which has a 36gb hard drive, but the IT company who supplied it assigned 1gb to /dev/root, 15mb to /dev/boot and 33gb to /dev/u. The /dev/root partition is now full, is there a way I can use the 33gb assigned to /dev/u without loosing any data, preferably... (2 Replies)
Discussion started by: Martyn
2 Replies
Login or Register to Ask a Question