CentOS 6 ran out of space, need to reclaim it


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers CentOS 6 ran out of space, need to reclaim it
# 22  
Old 08-27-2019
Great news and glad to hear you got it all sorted out.

Congrats and well done.

Well done Peasant and MIG - hitting the nail on the head. Boom!
# 23  
Old 08-27-2019
Dear DannyBoyCentOS,

My apologies, could you please repost your solution.

I made a mistake on mobile and accidentally edited your posts (and lost your info) instead of replying and quoting.

Was on mobile and made a mistake, my bad. Sorry.
# 24  
Old 08-27-2019
Quote:
Originally Posted by Neo
Dear DannyBoyCentOS,

My apologies, could you please repost your solution.

I made a mistake on mobile and accidentally edited your posts (and lost your info) instead of replying and quoting.

Was on mobile and made a mistake, my bad. Sorry.
Aw, shucks!

Peasant and MadeInGermany - thank you again, your solution worked.

Code:
umount /mnt/backup

cd /mnt/backup/
du -h
39G     ./servers-unix-hq/sugar.gnsa.local
39G     ./servers-unix-hq
4.5G    ./db-mysql-hq/sugar.gnsa.local
4.5G    ./db-mysql-hq
44G

Just like Peasant said, /mnt/backup was mounted on top of local directory with the same name. Once unmounted, the directory showed its true size.

Last edited by DannyBoyCentOS; 08-27-2019 at 01:19 PM.. Reason: readability
These 2 Users Gave Thanks to DannyBoyCentOS For This Post:
# 25  
Old 08-27-2019
To conclude check /etc/fstab for the mentioned mountpoint.

On first glance, a server was rebooted and came up without the mountpoint.
It happily worked until root filesystem is full, since same contains the service in question - mysql database.

At least, separate database filesystem as a logical volume, or even better a new disk partition with volume group.

A wise choice would be to separate /var inside root volume group, not now but for future projects Smilie

With seperate /var and mysql, you are very likely to survive full / partition without downtime.
In your case, only backup would not work, instead of making system almost unusable.

Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
# 26  
Old 08-27-2019
An additional note:

Under Linux there are SELinux and AppArmor around. They take care that services have only read/write/execute permissions which they need(predefined via config).

I do not know if they are armed at CentOS.

But if they are, file permissions are restricted to the configured mysql data location and one will get permission errors if moving the data directory without changing SELinux/AppArmor configuration as well.

---

What's regarding partition separation: I prefer all on the root-partition - as opposed to many recommendations especially from traditional unix administration point of view. My reason: It is simpler. I'm monitoring filespace, so it won't go undetected if the machines run low on space. I have a lot not so big machines. In another environment I might prefer seperated partitions.

What I recommend is to always use LVM. It's very flexible and you can move around filesystems to different disks online and can increase fs size online.

But to see the fact partition separation would have saved you in this case.

Last edited by stomp; 08-27-2019 at 02:04 PM..
# 27  
Old 08-27-2019
The main problem with putting all the data on the same root partition when running a production database is that if the database directory gets very large and fills up the root partition, it is possible the system will not boot and / or critical root partition log files will not log.

When running databases like we do with very large binary transactions logs, the database directory gets very large very fast.

However, when on a separate partition and full, not on root, the system will boot and continue to log (syslog, etc).

That is a big plus for most system admins knowing that a full data partition will not crash the boot partition.

For me, it's nothing to do with "traditional unix".. I do it because I do not want a full db partition to effect the boot partition and since I have seen this problem many times, it's better to have the db crash than the entire system / server crash.. I can log into a crash due to a full db partition, but I cannot login to a system which will not boot due to a full root / boot partition.

So, for me, all my data for a big database is off the root / boot partition, always (for production). Same is true for my backup directory of mysql dumps.

(As Peasant says as well in his analysis ...)
# 28  
Old 08-27-2019
Quote:
For me, it's nothing to do with "traditional unix"
What you wrote is part of what I meant: There are good reasons to do so. Yours and maybe others.... Just wanted to make it short for me.
This User Gave Thanks to stomp For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Safe way to shrink lvm vg_*-lv_swap partition and reclaim freed space on Linux?

Hello, # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 38.2G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 ... (2 Replies)
Discussion started by: centosadmin
2 Replies

2. AIX

How to reclaim hard disks and IP's in AIX?

Hello I recently received a request to reclaim hard disks and IP addresses within an AIX system(s). THe file systems are no longer in use and the client has indicated that it is OK to remove them and reclaim the disks and release the IP's. Now, since the file systems belong to a Volume group I... (8 Replies)
Discussion started by: Joseph Sabo
8 Replies

3. Red Hat

How to Upgrade Centos 5.7 using Centos 5.8 ISO image on Vmware workstation

Dear Linux Experts, On my windows 7 desktop with the help of Vmware workstation (Version 7.1), created virtual machine and installed Centos 5.7 successfully using ISO image. Query : Is this possible to upgrade the Centos 5.7 using Centos 5.8 ISO image to Centos version 5.8?.. if yes kindly... (2 Replies)
Discussion started by: Ananthcn
2 Replies

4. Red Hat

On CentOS, moving space from large free directory to another

Hi. My "/usr" folder is running out of space. My "/home" folder is quite large and has a lot of free space. As follows: Filesystem Type Size Used Avail Use% Mounted on ... /dev/sda5 ext3 9.7G 2.6G 6.7G 28% / /dev/sda7 ext3 152G 16G 128G 11% /home /dev/sda3 ... (7 Replies)
Discussion started by: pkiula
7 Replies

5. Linux

How to reclaim the space which i used to increse the swap space on Xen,

Hi, i have done a blunder here, i increased the swap space on Xen5.6 server machine using below steps :- 1056 dd if=/dev/zero of=/root/myswapfile bs=1M count=1024 1057 ls -l /root/myswapfile 1058 chmod 600 /root/myswapfile 1059 mkswap /root/myswapfile 1060 swapon /root/myswapfile ... (1 Reply)
Discussion started by: apm
1 Replies

6. Shell Programming and Scripting

how to find whether a script ran or not

Hi, I have written a script and placed in an application and the script can be executed manually only. But somehow one of the method in the script is being called and bringing the application down. But we are not able to find any instance of script running. Is there a way to findout whether the... (1 Reply)
Discussion started by: Satyak
1 Replies

7. Shell Programming and Scripting

Identify if ran by su or sudo?

Recently I was on an operational call and heard the people running my code placing the code in the /tmp directory and running as root. I had not planned on that. So I want to add some checks to my code (using ksh93): # ---------- ---------- ---------- # root not allowed to run this #... (3 Replies)
Discussion started by: ericdp63
3 Replies

8. UNIX for Dummies Questions & Answers

Reclaim deleted disk space

I have a disk space issue on one of my unix servers. it is showing 98% full.. i found the offending folder and removed it. but i have not reclaimed the disk space. is there another command that i need ? thank you in advance for any assistance. (4 Replies)
Discussion started by: JanSP
4 Replies

9. Shell Programming and Scripting

which user ran which command

can we come to know all the command ran bya user for last 1 day (1 Reply)
Discussion started by: narang.mohit
1 Replies

10. Filesystems, Disks and Memory

Ran out of space on /dev/root partition

hi, I have a SCO unix server which has a 36gb hard drive, but the IT company who supplied it assigned 1gb to /dev/root, 15mb to /dev/boot and 33gb to /dev/u. The /dev/root partition is now full, is there a way I can use the 33gb assigned to /dev/u without loosing any data, preferably... (2 Replies)
Discussion started by: Martyn
2 Replies
Login or Register to Ask a Question