Visit Our UNIX and Linux User Community


CentOS 6 ran out of space, need to reclaim it


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers CentOS 6 ran out of space, need to reclaim it
# 1  
Old 08-26-2019
CentOS 6 ran out of space, need to reclaim it

Hello everyone,

I am having an issue here with CentOS release 6.6 (Final) that shows all of the space used up, but I can't tell where the space went.

Seemingly I am using up 100%, according to

Code:
df -h

Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_sugar-lv_root   50G   49G     0 100% /

So that's 49G out of 50G, fair enough, but when I try to see what exactly is being used (x to remove mounts):

Code:
du -xsh /*

1.8G    /usr
2.0G    /var

That's all, these are the 2 biggest directories, all others combined are under 1G.

Here is info on Disk /dev/mapper/vg_sugar-lv_root, it does show close to 50G (this is a virtual machine):
Code:
fdisk -l

Disk /dev/mapper/vg_sugar-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Node count is at 7%:
Code:
df -i

Filesystem                   Inodes  IUsed     IFree   IUse% Mounted on
/dev/mapper/vg_sugar-lv_root 3276800 200552    3076248    7% /

The system was rebooted several times, so any log files should've been deleted. I'd appreciate some tips in resolving this.

Code:
find / -xdev -type f -size +100M

returned only a single file. The problem is not in amount of large files, but that the space is hidden or used for something else, or not released.
Code:
 du -ahx / | sort -rh | head -20 
5.0G / 
2.0G /var 
1.8G /usr
1.1G /var/lib 
945M /var/lib/mysql 
909M /var/lib/mysql/sugarcrm 
759M /var/www 
745M /var/www/html 
707M /root 
672M /usr/lib64 
642M /var/www/html/sugar 
611M /usr/share 
480M /var/lib/mysql/sugarcrm/emails_text.MYD 
441M /lib 
392M /lib/modules 

228M /usr/lib


Last edited by Neo; 09-19-2019 at 02:00 AM.. Reason: Added icode tags to discussion
# 2  
Old 08-26-2019
Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.
# 3  
Old 08-26-2019
Refine your search for smaller files, recently updated:
Code:
find / -xdev -type f -size +1000 -mtime -60

Search for deleted files:
Code:
lsof / | grep "(deleted)"

# 4  
Old 08-26-2019
> Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.

What does, then?

--- Post updated at 06:47 PM ---

Quote:
Originally Posted by Neo
Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.
What steps should I take to delete log files or reclaim the space, then?

--- Post updated at 07:02 PM ---

Quote:
Originally Posted by MadeInGermany
Refine your search for smaller files, recently updated:
Code:
find / -xdev -type f -size +1000 -mtime -60

Search for deleted files:
Code:
lsof / | grep "(deleted)"

Code:
lsof / | grep "(deleted)"

returned no entries
Code:
lsof | grep -i del

shows file on /dev/zero only

Code:
find / -xdev -type f -size +1000 -mtime -60

returned some entries

Code:
-rw-r--r--  1 root   root    11M Aug 23 02:03 sugarcrm_1.log
-rw-r--r--  1 root   root    11M Aug 19 12:38 sugarcrm_2.log
-rw-r--r--  1 root   root    11M Aug 16 02:03 sugarcrm_3.log
-rw-r--r--  1 root   root    11M Aug 12 18:04 sugarcrm_4.log
-rw-r--r--  1 root   root    11M Aug 11 05:05 sugarcrm_5.log
-rw-r--r--  1 root   root    11M Aug  9 22:13 sugarcrm_6.log
-rw-r--r--  1 root   root    11M Aug  8 12:22 sugarcrm_7.log
-rw-r--r--  1 root   root    11M Aug  7 05:35 sugarcrm_8.log
-rw-r--r--  1 root   root    11M Aug  5 10:19 sugarcrm_9.log

I deleted those files, freed up 100 Mbs, but the MySQL is not starting, reporting
190826 11:00:55 InnoDB: Started; log sequence number 0 8747645
190826 11:00:55 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190826 11:00:55 [ERROR] Can't start server: can't create PID file: No space left on device
# 5  
Old 08-26-2019
Ensure with
Code:
ls /var/run/mysqld
df /var/run/mysqld

that it is empty and on the / filesystem.
A command to find the biggest directories in the / file system:
Code:
find / -xdev -mindepth 2 -maxdepth 2 -exec du -skx {} + | sort -k1,1n

The following has no precise space count but ensures to not enter other filesystems:
Code:
find / -depth -xdev -ls | awk '{ cnt+=int($7/1024); if (split ($11,A,"/")==3) { print cnt, $0; cnt=0 } }' | sort -k1,1n

# 6  
Old 08-26-2019
Quote:
Originally Posted by MadeInGermany
Ensure with
Code:
ls /var/run/mysqld
df /var/run/mysqld

that it is empty and on the / filesystem.
A command to find the biggest directories in the / file system:
Code:
find / -xdev -mindepth 2 -maxdepth 2 -exec du -skx {} + | sort -k1,1n

The following has no precise space count but ensures to not enter other filesystems:
Code:
find / -depth -xdev -ls | awk '{ cnt+=int($7/1024); if (split ($11,A,"/")==3) { print cnt, $0; cnt=0 } }' | sort -k1,1n

I made sure that ls /var/run/mysqld is empty, and df /var/run/mysqld is on /
Code:
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                      51475068 50220120         0 100% /

Code:
find / -xdev -mindepth 2 -maxdepth 2 -exec du -skx {} + | sort -k1,1n
51164   /root/sugarcrm_full-2014-01-29T01-30-01.sql
77020   /root/sugarcrm_sqldump_2014_07_08.sql
80240   /root/SugarCE-Full-6.5.4
86672   /root/sugar_backup
90268   /var/log
152124  /root/sugar_6_4_backup
163828  /usr/bin
206900  /root/sugar_07_10_2013
232744  /usr/lib
400408  /lib/modules
625660  /usr/share
684664  /var/www
687556  /usr/lib64
1062204 /var/lib
13812205161     /mnt/backup

Is it normal for /mnt/backup to show? I thought -x in xdev would exclude external file systems?

Code:
df /mnt
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                      51475068 50220300         0 100% /

Code:
find / -depth -xdev -ls | awk '{ cnt+=int($7/1024); if (split ($11,A,"/")==3) { print cnt, $0; cnt=0 } }' | sort -k1,1n

I've attached find.txt to this message with the output.
# 7  
Old 08-27-2019
DannyBoyCentOS,

You will benefit from thinking about what you are saying about log files.

Code:
Neo: Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.

DannyBoyCentOS:  What does, then?

Any system which would automatically delete log files on reboot would be a seriously misconfigured system.

Log files are the single most important tool any system administrator has to deal with configuration errors, security breaches, system errors, access control issues, database errors, and more.

Without logging "all is lost".... as one might say, so no normally configured Linux system automatically deletes a single log file during the boot process. When you post like this, an experienced Linux user like me (over two decades of Linux system admin) can only conclude you have very little experience on a server. So let me explain to you again.

Log files should never be automatically deleted on reboot because that means anytime the system reboots, all that logging information would be lost; and a server can reboot for a variety of reasons. Log files are generally rotated and compressed and saved by the system; and then they are generally archived for a certain period of time, and then eventually deleted manually. The could be deleted automatically after some set period of time, but that should normally be a few weeks out (actually it depends on the size of storage, system criticality, system usages and other site specific factors).

In your output, the big files I see are (mysql) dump files and backups. These are not log files, they are backups and dumps. Normally these files can get very big very quickly and need to be actively managed. I manage these on my servers by moving them to an archive site (generally another servers for that specific server); but how you choose to do that is entirely up to you.

Based on what I have seen in your output so far (which admittedly I did not spend too much time on it), the big files you have are .sql and .zip files. If you need disk space, move these to another disk, another server, the cloud or where ever you archive your large files, dumps and backups; and then decide what to keep and what to delete.

Cheers.

Previous Thread | Next Thread
Test Your Knowledge in Computers #914
Difficulty: Medium
In bash, grep -v -e '^[[:space:]]*$' file.txt will filter all the blank lines from file.txt and save the results in a file called file.txt.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Safe way to shrink lvm vg_*-lv_swap partition and reclaim freed space on Linux?

Hello, # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 38.2G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 ... (2 Replies)
Discussion started by: centosadmin
2 Replies

2. AIX

How to reclaim hard disks and IP's in AIX?

Hello I recently received a request to reclaim hard disks and IP addresses within an AIX system(s). THe file systems are no longer in use and the client has indicated that it is OK to remove them and reclaim the disks and release the IP's. Now, since the file systems belong to a Volume group I... (8 Replies)
Discussion started by: Joseph Sabo
8 Replies

3. Red Hat

How to Upgrade Centos 5.7 using Centos 5.8 ISO image on Vmware workstation

Dear Linux Experts, On my windows 7 desktop with the help of Vmware workstation (Version 7.1), created virtual machine and installed Centos 5.7 successfully using ISO image. Query : Is this possible to upgrade the Centos 5.7 using Centos 5.8 ISO image to Centos version 5.8?.. if yes kindly... (2 Replies)
Discussion started by: Ananthcn
2 Replies

4. Red Hat

On CentOS, moving space from large free directory to another

Hi. My "/usr" folder is running out of space. My "/home" folder is quite large and has a lot of free space. As follows: Filesystem Type Size Used Avail Use% Mounted on ... /dev/sda5 ext3 9.7G 2.6G 6.7G 28% / /dev/sda7 ext3 152G 16G 128G 11% /home /dev/sda3 ... (7 Replies)
Discussion started by: pkiula
7 Replies

5. Linux

How to reclaim the space which i used to increse the swap space on Xen,

Hi, i have done a blunder here, i increased the swap space on Xen5.6 server machine using below steps :- 1056 dd if=/dev/zero of=/root/myswapfile bs=1M count=1024 1057 ls -l /root/myswapfile 1058 chmod 600 /root/myswapfile 1059 mkswap /root/myswapfile 1060 swapon /root/myswapfile ... (1 Reply)
Discussion started by: apm
1 Replies

6. Shell Programming and Scripting

how to find whether a script ran or not

Hi, I have written a script and placed in an application and the script can be executed manually only. But somehow one of the method in the script is being called and bringing the application down. But we are not able to find any instance of script running. Is there a way to findout whether the... (1 Reply)
Discussion started by: Satyak
1 Replies

7. Shell Programming and Scripting

Identify if ran by su or sudo?

Recently I was on an operational call and heard the people running my code placing the code in the /tmp directory and running as root. I had not planned on that. So I want to add some checks to my code (using ksh93): # ---------- ---------- ---------- # root not allowed to run this #... (3 Replies)
Discussion started by: ericdp63
3 Replies

8. UNIX for Dummies Questions & Answers

Reclaim deleted disk space

I have a disk space issue on one of my unix servers. it is showing 98% full.. i found the offending folder and removed it. but i have not reclaimed the disk space. is there another command that i need ? thank you in advance for any assistance. (4 Replies)
Discussion started by: JanSP
4 Replies

9. Shell Programming and Scripting

which user ran which command

can we come to know all the command ran bya user for last 1 day (1 Reply)
Discussion started by: narang.mohit
1 Replies

10. Filesystems, Disks and Memory

Ran out of space on /dev/root partition

hi, I have a SCO unix server which has a 36gb hard drive, but the IT company who supplied it assigned 1gb to /dev/root, 15mb to /dev/boot and 33gb to /dev/u. The /dev/root partition is now full, is there a way I can use the 33gb assigned to /dev/u without loosing any data, preferably... (2 Replies)
Discussion started by: Martyn
2 Replies

Featured Tech Videos