Sponsored Content
Operating Systems Linux Debian Problem with maximum files and directories Post 302952938 by ganon551 on Monday 24th of August 2015 12:10:18 PM
Old 08-24-2015
Thank you for answering.

I should have post some code, that would have make things easier for sure.

I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.

I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.

I also switched to perl as I thought it would be faster, but even with that the writing takes ages.

Here's some code :

Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>)  {   
    $line =~ s/\r|\n//g;
    my $md5 = md5_hex($line);
    my $add = substr $md5,0,1;
    my $add2 = substr $md5,1,3;
    my $add3 = substr $md5,4,3;
    $outfile = "md5hash__/".$add."/md5".$add2.".txt";
        open (FILE, ">> $outfile") || die "problem opening $outfile\n";
    print FILE $add3."\n".$line."\n";
    close(FILE);
}

Don't mind the code, it was just to try and see the speed.

Thank you for your help Smilie

EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.

Last edited by ganon551; 08-24-2015 at 01:23 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Maximum files in a Directory

Does Solaris impose limits on : - the maximum number of files a directory can have, - total file size in a directory If there is such limits, how can I can check for each? Thanks...:confused: (1 Reply)
Discussion started by: deaniyoer
1 Replies

2. UNIX for Dummies Questions & Answers

ls - maximum number of files

what is the maximum number ls can list down (6 Replies)
Discussion started by: karnan
6 Replies

3. Linux

maximum file & directories

Dear friends, How many maximum number of files and directories can be created in linux system. Thanks.. (2 Replies)
Discussion started by: karthigayan
2 Replies

4. UNIX for Dummies Questions & Answers

List directories and sub directories recursively excluding files

Hi, Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process. I wanted to disply using a unix command all the directories recursively excluding files. I tried 'ls -FR' but that display files as... (3 Replies)
Discussion started by: pointers
3 Replies

5. Shell Programming and Scripting

Sub directories containing maximum files

Hi All, I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain. find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10 But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Discussion started by: adurga
5 Replies

6. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

7. Shell Programming and Scripting

Rename the files in all the directories and sub-directories

Hi all, I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”. I need to... (1 Reply)
Discussion started by: AshwaniSharma09
1 Replies

8. Shell Programming and Scripting

Extracting files having maximum timestamp

Hi , I'm using Ksh 88 I've the following files in a directory with YearMonthDate (Ex:20130601) YDT:FILE1:20130601 YDT:FILE1:20130615 YDT:FILE2:20130601 YDT:FILE2:20130615 YDT:FILE3:20130601 YDT:FILE3:20130615 And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
Discussion started by: smile689
8 Replies

9. Shell Programming and Scripting

Archiving and moving files into directories, creating directories, etc.

how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex. so mydemo--2015-03-23-1500.tar.gz > database -... (5 Replies)
Discussion started by: wyclef
5 Replies
File(3pm)						User Contributed Perl Documentation						 File(3pm)

NAME
Digest::MD5::File - Perl extension for getting MD5 sums for files and urls. SYNOPSIS
use Digest::MD5::File qw(dir_md5_hex file_md5_hex url_md5_hex); my $md5 = Digest::MD5->new; $md5->addpath('/path/to/file'); my $digest = $md5->hexdigest; my $digest = file_md5($file); my $digest = file_md5_hex($file); my $digest = file_md5_base64($file); my $md5 = Digest::MD5->new; $md5->addurl('http://www.tmbg.com/tour.html'); my $digest = $md5->hexdigest; my $digest = url_md5($url); my $digest = url_md5_hex($url); my $digest = url_md5_base64($url); my $md5 = Digest::MD5->new; $md5->adddir('/directory'); my $digest = $md5->hexdigest; my $dir_hashref = dir_md5($dir); my $dir_hashref = dir_md5_hex($dir); my $dir_hashref = dir_md5_base64($dir); DESCRIPTION
Get MD5 sums for files of a given path or content of a given url. EXPORT
None by default. You can export any file_* dir_*, or url_* function and anything Digest::MD5 can export. use Digest::MD5::File qw(md5 md5_hex md5_base64); # 3 Digest::MD5 functions print md5_hex('abc123'), " "; print md5_base64('abc123'), " "; OBJECT METHODS
addpath() my $md5 = Digest::MD5->new; $md5->addpath('/path/to/file.txt') or die "file.txt is not where you said: $!"; or you can add multiple files by specifying an array ref of files: $md5->addpath(@files); adddir() addpath()s each file in a directory recursively. Follows the same rules as the dir_* functions. my $md5 = Digest::MD5->new; $md5->adddir('/home/tmbg/') or die "See warning above to see why I bailed: $!"; addurl() my $md5 = Digest::MD5->new; $md5->addurl('http://www.tmbg.com/tour.html') or die "They Must Be not on tour"; file_* functions Get the digest in variouse formats of $file. If file does not exist or is a directory it croaks (See NOFATALS for more info) my $digest = file_md5($file) or warn "$file failed: $!"; my $digest = file_md5_hex($file) or warn "$file failed: $!"; my $digest = file_md5_base64($file) or warn "$file failed: $!"; dir_* functions Returns a hashref whose keys are files relative to the given path and the values are the MD5 sum of the file or and empty string if a directory. It recurses through the entire depth of the directory. Symlinks to files are just addpath()d and symlinks to directories are followed. my $dir_hashref = dir_md5($dir) or warn "$dir failed: $!"; my $dir_hashref = dir_md5_hex($dir) or warn "$dir failed: $!"; my $dir_hashref = dir_md5_base64($dir) or warn "$dir failed: $!"; url_* functions Get the digest in various formats of the content at $url (Including, if $url points to directory, the directory listing content). Returns undef if url fails (IE if LWP::UserAgent's $res->is_success is false) my $digest = url_md5($url) or warn "$url failed"; my $digest = url_md5_hex($url) or warn "$url failed"; my $digest = url_md5_base64($url) or warn "$url failed"; SPECIAL SETTINGS
BINMODE By default files are opened in binmode. If you do not want to do this you can unset it a variety of ways: use Digest::MD5::File qw(-nobin); or $Digest::MD5::File::BINMODE = 0; or at the function/method level by specifying its value as the second argument: $md5->addpath($file,0); my $digest = file_md5_hex($file,0); UTF8 In some cases you may want to have your data utf8 encoded, you can do this the following ways: use Digest::MD5::File qw(-utf8); or $Digest::MD5::File::UTF8 = 1; or at the function/method level by specifying its value as the third argument for files and second for urls: $md5->addpath($file,$binmode,1); my $digest = file_md5_hex($file,$binmode,1); $md5->addurl($url,1); url_md5_hex($url,1); It use's Encode's encode_utf8() function to do the encoding. So if you do not have Encode (pre 5.7.3) this won't work :) NOFATALS Instead of croaking it will return undef if you set NOFATALS to true. You can do this two ways: $Digest::MD5::File::NOFATALS = 1; or the -nofatals flag: use Digest::MD5::File qw(-nofatals); my $digest = file_md5_hex($file) or die "$file failed"; $! is not set so its not really helpful if you die(). SEE ALSO
Digest::MD5, Encode, LWP::UserAgent AUTHOR
Daniel Muey, <http://drmuey.com/cpan_contact.pl> COPYRIGHT AND LICENSE
Copyright 2005 by Daniel Muey This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.14.2 2012-04-04 File(3pm)
All times are GMT -4. The time now is 01:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy