Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Delete big directory issue

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete big directory issue
# 8  
Old 01-15-2014
Quote:
Originally Posted by Corona688
If it's taking 11 hours to delete a directory, it's either a very, very full directory, or rm is deleting things you didn't intend.

If it's just a very full directory, rm doesn't have a "go faster" button for that. It's the filesystem that's slow, from having far too many files piled into one folder.


Actual directory structure is like that.

Code:
/2014-01-14/0-1/dir1
/2014-01-14/0-1/dir2
/2014-01-14/0-1/dir3

/2014-01-14/1-2/dir1
/2014-01-14/1-2/dir2
/2014-01-14/1-2/dir3

/2014-01-14/21-22/dir1
/2014-01-14/21-22/dir2
/2014-01-14/21-22/dir3

script is deleting 2014-01-14 so it is taking alot of time, total size of one day old log equal to 70GB around and each hour directory size is aound 2.3GB.

---------- Post updated at 02:39 PM ---------- Previous update was at 02:38 PM ----------

Quote:
Originally Posted by Scott
Are there any network-mounted directories in there?
Basically is SAN storage assign disk.
# 9  
Old 01-15-2014
Maybe you should consider putting that directory on its own filesystem, then you can delete it in no time at all.
# 10  
Old 01-15-2014
Scott raises a good point about network mounted directories. If there's any issues with those, that could be a bottleneck.

Nonetheless, it's not rm that's misbehaving here.
# 11  
Old 01-15-2014
On operating system have not much space, we have only option for SAN. Is there any possibility to use ionice or nice with cron so it will take low cpu/io utilization and not create load issue on server.
# 12  
Old 01-15-2014
The /bin/rm binary should be the fastest.
Please state your OS and file system type!
Code:
uname
df $date

# 13  
Old 01-15-2014
Quote:
Originally Posted by MadeInGermany
The /bin/rm binary should be the fastest.
Please state your OS and file system type!
Code:
uname
df $date

Code:
1) Linux RHEL5
2) ext3
3) kernel 2.6
3) currently i am
4) Filesystem is 70% available disk space.

---------- Post updated at 02:54 PM ---------- Previous update was at 02:51 PM ----------

Please check below link, is it possible we can do something for directories too..

How to delete million of files on busy Linux servers (Work out Argument list too long) | Walking in Light with Christ - Faith, Computing, Diary
# 14  
Old 01-15-2014
That isn't relevant. You aren't getting the 'too many arguments' error. That won't solve any problems you're actually having.

I know this isn't what you want to hear, but the best way to handle this problem would have been preventatively -- by not storing your files in such an unwieldy way.

rm doesn't have a "go faster" button. To make it work faster you need faster disks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

Big /etc/group creating issue

I have Solaris-10 with kernel patch 142900-03, Update 7 Sparc server. root@ddlomps40:/# uname -a SunOS ddlomps40 5.10 Generic_142900-03 sun4u sparc SUNW,SPARC-Enterprise root@ddlomps40:/# cat /etc/release Solaris 10 5/09 s10s_u7wos_08 SPARC Copyright 2009 Sun... (5 Replies)
Discussion started by: solaris_1977
5 Replies

2. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

3. UNIX for Advanced & Expert Users

Delete first 100 lines from a BIG File

Hi, I need a unix command to delete first n (say 100) lines from a log file. I need to delete some lines from the file without using any temporary file. I found sed -i is an useful command for this but its not supported in my environment( AIX 6.1 ). File size is approx 100MB. Thanks in... (18 Replies)
Discussion started by: unohu
18 Replies

4. Shell Programming and Scripting

Delete rows from big file

Hi all, I have a big file (about 6 millions rows) and I have to delete same occurrences, stored in a small file (about 9000 rews). I have tried this: while read line do grep -v $line big_file > ok_file.tmp mv ok_file.tmp big_file done < small_file It works, but is very slow. How... (2 Replies)
Discussion started by: Tibbeche
2 Replies

5. Shell Programming and Scripting

Hello - Have a big issue need help please ASAP

I have been using this simple grep command to extract fields from a file that ALPHANUMERIC field has been set to 16 grep -w 'ALPHANUMERIC *16' filename ------------------------- sample data file SAMP_RICS "SAMPLE RICS" 5694 NULL ALPHANUMERIC 60 RMTES_STRING ... (11 Replies)
Discussion started by: mnassiri
11 Replies

6. HP-UX

Empty Directory, big size

Hello, Can you please explain why I have various empty directories with large size ? OS is B.11.11 (3 Replies)
Discussion started by: drbiloukos
3 Replies

7. UNIX for Dummies Questions & Answers

Empty Directory, big size

Hello, Can somebody please explain why I have EMPTY directories on HP-UX with BIG SIZE ? Thank you ! Double post, continued here (0 Replies)
Discussion started by: drbiloukos
0 Replies

8. Shell Programming and Scripting

Find which dir is big size in a directory

Hi all, Could you please tellme the commadn which sorts the list of directories in a parent dir by their size. Thanks. (2 Replies)
Discussion started by: firestar
2 Replies

9. HP-UX

Processing very big directory

Hello, I have a very big directory with lots of small files. Just doing a single "ls" takes eternity, so I even can't know how many files the directory has. I need to get the newest files in the last 15 minutes, but I can't get an eficient way of doing it. I've tried find, perl, shell script,... (4 Replies)
Discussion started by: psimoes79
4 Replies

10. Shell Programming and Scripting

Need script to make big directory structure

Hi, I'm a novice and I'd like to make a directory structure with a hundred or so folders. I've tried mkdir /foo/foo1/etc... mkdir /foo/foo2/etc mkdir /foo/foo3/etc mkdir /foo/foo4/etc ...but it appends '@' to each folder name and then fails on the subdirectories. Is it better to use a... (2 Replies)
Discussion started by: kamur
2 Replies
Login or Register to Ask a Question