Delete big directory issue


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete big directory issue
# 15  
Old 01-15-2014
If $date is a separate file system, you can consider umount, mkfs, mount.
# 16  
Old 01-15-2014
Is it possible i can run below command on subdirectories in someway and make some loop, but don't understand the logic.

Code:
time perl -e 'for(<*>){((stat)[9]<(unlink))}'

---------- Post updated at 03:08 PM ---------- Previous update was at 03:07 PM ----------

Quote:
Originally Posted by MadeInGermany
If $date is a separate file system, you can consider umount, mkfs, mount.
Its new server, already mounted file system 2 days ago.
# 17  
Old 01-15-2014
Quote:
Originally Posted by learnbash
Is it possible i can run below command on subdirectories in someway and make some loop, but don't understand the logic.
This isn't faster than rm. rm is not "running slow" here. If you wrote a super-efficient delete program in three assembly-language instructions, it'd still be slow.

The problem is your directory. Every time you delete a file, the kernel has to update the list of files. When the list of files is really big, this can take a long time.

I know this isn't what you want to hear but there's no "delete faster" button.
# 18  
Old 01-15-2014
file sizes are in kilobytes, that is txt files
# 19  
Old 01-15-2014
The issue, as stated before, is the number of files not their sizes.

You said directories have about 2.3G of data and files are kilobytes in size so I'm guessing each directory my have 100,000 or more files.

Unix filesystems are not designed to have such large numbers of files in a single directory and performance is suffering.

Last edited by Chubler_XL; 01-15-2014 at 04:41 PM.. Reason: Spelling
# 20  
Old 01-15-2014
how can i run below program in one line. I need to test that.

Code:
use strict;
use warnings;
use File::Find;
my $dir = "/data/2014-01-14/";
find(\&wanted, $dir);
sub wanted {
unlink glob "*.xml";
}

# 21  
Old 01-15-2014
Quote:
Originally Posted by learnbash
Is it possible i can run below command on subdirectories in someway and make some loop, but don't understand the logic.

Code:
time perl -e 'for(<*>){((stat)[9]<(unlink))}'

---------- Post updated at 03:08 PM ---------- Previous update was at 03:07 PM ----------



Its new server, already mounted file system 2 days ago.
And it was empty, wasn't it?
Redo it, and it will be empty again...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

Big /etc/group creating issue

I have Solaris-10 with kernel patch 142900-03, Update 7 Sparc server. root@ddlomps40:/# uname -a SunOS ddlomps40 5.10 Generic_142900-03 sun4u sparc SUNW,SPARC-Enterprise root@ddlomps40:/# cat /etc/release Solaris 10 5/09 s10s_u7wos_08 SPARC Copyright 2009 Sun... (5 Replies)
Discussion started by: solaris_1977
5 Replies

2. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

3. UNIX for Advanced & Expert Users

Delete first 100 lines from a BIG File

Hi, I need a unix command to delete first n (say 100) lines from a log file. I need to delete some lines from the file without using any temporary file. I found sed -i is an useful command for this but its not supported in my environment( AIX 6.1 ). File size is approx 100MB. Thanks in... (18 Replies)
Discussion started by: unohu
18 Replies

4. Shell Programming and Scripting

Delete rows from big file

Hi all, I have a big file (about 6 millions rows) and I have to delete same occurrences, stored in a small file (about 9000 rews). I have tried this: while read line do grep -v $line big_file > ok_file.tmp mv ok_file.tmp big_file done < small_file It works, but is very slow. How... (2 Replies)
Discussion started by: Tibbeche
2 Replies

5. Shell Programming and Scripting

Hello - Have a big issue need help please ASAP

I have been using this simple grep command to extract fields from a file that ALPHANUMERIC field has been set to 16 grep -w 'ALPHANUMERIC *16' filename ------------------------- sample data file SAMP_RICS "SAMPLE RICS" 5694 NULL ALPHANUMERIC 60 RMTES_STRING ... (11 Replies)
Discussion started by: mnassiri
11 Replies

6. HP-UX

Empty Directory, big size

Hello, Can you please explain why I have various empty directories with large size ? OS is B.11.11 (3 Replies)
Discussion started by: drbiloukos
3 Replies

7. UNIX for Dummies Questions & Answers

Empty Directory, big size

Hello, Can somebody please explain why I have EMPTY directories on HP-UX with BIG SIZE ? Thank you ! Double post, continued here (0 Replies)
Discussion started by: drbiloukos
0 Replies

8. Shell Programming and Scripting

Find which dir is big size in a directory

Hi all, Could you please tellme the commadn which sorts the list of directories in a parent dir by their size. Thanks. (2 Replies)
Discussion started by: firestar
2 Replies

9. HP-UX

Processing very big directory

Hello, I have a very big directory with lots of small files. Just doing a single "ls" takes eternity, so I even can't know how many files the directory has. I need to get the newest files in the last 15 minutes, but I can't get an eficient way of doing it. I've tried find, perl, shell script,... (4 Replies)
Discussion started by: psimoes79
4 Replies

10. Shell Programming and Scripting

Need script to make big directory structure

Hi, I'm a novice and I'd like to make a directory structure with a hundred or so folders. I've tried mkdir /foo/foo1/etc... mkdir /foo/foo2/etc mkdir /foo/foo3/etc mkdir /foo/foo4/etc ...but it appends '@' to each folder name and then fails on the subdirectories. Is it better to use a... (2 Replies)
Discussion started by: kamur
2 Replies
Login or Register to Ask a Question