Visit The New, Modern Unix Linux Community


Issue with rm command in a tera byte file system


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Issue with rm command in a tera byte file system
# 1  
Issue with rm command in a tera byte file system

We have a production file system which has 6+ million files with more than 1 tera byte in size. When trying to delete selective files through a weekly script files are not deleted.

Please advise with ideas.
# 2  
What is the output from ls -l file... for the files you want to delete?

What is the output from ls -ld directory for the directory (or directories) containing the file(s) that you couldn't delete?

What diagnostics did rm file... produce for the files that were not removed?

What user and groups privileges are being used when the script is running? (I.e., what is the output from the command id?

What shell is being used to run the script that is failing? What are the exact commands in the script that are failing? What operating system (including version) are you using?
# 3  
Is this cron scheduled and if so does your script get called properly, setting up the appropriate environment before it starts doing active work?
# 4  
I would guess that an rm request takes so long that you interpret the wait as not working - assuming that the job that deletes files 'worked' before.

Some filesystems do not perform well on super huge numbers of files in directories, and response times can literally be an hour for a simple
Code:
ls /path/to/files/*.extension

command.
# 5  
Facebook

For starters please post the script and mention what Operating System and Shell you have.

At a guess your script generates a command which is too long for the Operating System.

Bearing in mind that we know nothing about your Operating System or Shell, the generic solution format is something like this (in ksh) where the criteria is to delete files more than 180 days old :

Code:
find /cullable_tree/ -type f -mtime +180 -print | while read FILENAME
do
     if [ -f "${FILENAME}" ]
     then
              echo "${FILENAME}"
              # rm "${FILENAME}"
              # (Uncomment when tested thoroughly)
     fi
done

The "if -f" covers circumstances where the filename contains weird characters.

This sort of construct never generates long command lines and may not be fast but it gets there in the end.
# 6  
You could shorten this to a single-line find command, but can you (kppublicmail) answer the questions posed in responses. It will mean that you get more useful suggestions.

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #415
Difficulty: Easy
HTML, CSS, and JavaScript are the three core technologies of the World Wide Web.
True or False?

10 More Discussions You Might Find Interesting

1. HP-UX

Traditional Chinese Multi-Byte issue

Trying to load a specific Traditional Chinese Character on an HP UX platform for a Taiwan database. Data is converting to ?. Database has existed since 2008 and this is the first issue I've seen where a user is unable to save the multi-byte character in the database. I'm thinking this is may be... (0 Replies)
Discussion started by: OHcoder
0 Replies

2. Red Hat

Issue with mounting the file system .

Hi gurus, Need your help , Am facing some issue with one of the ext3 file system.while rebooting it failed to mount fs after running fsck i tried manuall but still no luck.However i made the machine up but am unable to mount file system can some one please help me. root@vm258902]~# cat... (2 Replies)
Discussion started by: kapil514
2 Replies

3. Shell Programming and Scripting

Shared File system- lib access issue

I have a C++ binary executable installed in a file system which is shared across multiple solaris boxes. When I start this executable from one of the boxes,I am able to start only 4 parallel instances and from the 5th instance onwards I am getting the following error. fatal: libdb2.so.1:... (2 Replies)
Discussion started by: prasperl
2 Replies

4. AIX

GPFS file system corrupt issue

Hi Administrators, I have one issue related to GPFS filesystem. We have bad entries in this file system, which reflects error like ls: 0653-341 The file <filename> does not exist. when we give "ls -ltr" on this directory. So we taken the FS filesystem offline and followed the below steps. ... (6 Replies)
Discussion started by: gauravgarg
6 Replies

5. Solaris

/ file system full issue

Hi All, This is Babu working as a system administrator. Here I am getting one problem with one of my Sun server's root (/) file system. In df -h command / file system showing 7.8 GB used space.But in du -hd command it showing 5.2 gb only. Please can any one help me resolve this issue... (2 Replies)
Discussion started by: lbreddy
2 Replies

6. Solaris

root file system issue

Hi, In df -h root file system showing (total size) (used) (free) /dev/dsk/c0t0d0s0 11G 7.6G 2.8G 73% / but du -hd command showng 5.1 gb used sapce. I am missing 3.00 gb space. Here I have to knoe where is Remaining space... (2 Replies)
Discussion started by: lbreddy
2 Replies

7. Solaris

File system Issue

Hi All, I am working with the Solaris 9. In df -h command / file system size is 9.6 GB , used 7.5 GB and avialble space is 2.1. OK But in du -hd command it is showing some 4.1 GB only. How to find out that remaining 3.4GB. Can you please help me ... Thanks and Regards, (1 Reply)
Discussion started by: lbreddy
1 Replies

8. Shell Programming and Scripting

Check if 2 files are identical byte-to-byte?

In my server migration requirement, I need to compare if one file on old server is exactly the same as the corresponding file on the new server. For diff and comm, the inputs need to be sorted. But I do not want to disturb the content of the file and need to find byte-to-byte match. Please... (4 Replies)
Discussion started by: krishmaths
4 Replies

9. Solaris

Tera Terminal Emulator

Hello Expert! :b: Question for you guys, Can anyone tell me how to use terminal emulator on Windows XP to view Solaris config? I have no idea on Solaris and the only thing I could do is to boot it up. Honestly, I have given a tasked to delete all the files and some necessary memory information... (2 Replies)
Discussion started by: katsloko
2 Replies

10. Shell Programming and Scripting

byte count and cut command

1. Is there a way to count the number of bytes of a variable? example: abc@yahoo.com is 13 bytes 2. Cut command only allows one byte for delimiter example: cut -f1 -d'.' delimited by period. Is there a way to have two or more characters in the delimiter field? thanks in adavance. :) (4 Replies)
Discussion started by: hemangjani
4 Replies

Featured Tech Videos