Sponsored Content
Operating Systems Linux Debian Problem with maximum files and directories Post 302952938 by ganon551 on Monday 24th of August 2015 12:10:18 PM
Old 08-24-2015
Thank you for answering.

I should have post some code, that would have make things easier for sure.

I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.

I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.

I also switched to perl as I thought it would be faster, but even with that the writing takes ages.

Here's some code :

Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>)  {   
    $line =~ s/\r|\n//g;
    my $md5 = md5_hex($line);
    my $add = substr $md5,0,1;
    my $add2 = substr $md5,1,3;
    my $add3 = substr $md5,4,3;
    $outfile = "md5hash__/".$add."/md5".$add2.".txt";
        open (FILE, ">> $outfile") || die "problem opening $outfile\n";
    print FILE $add3."\n".$line."\n";
    close(FILE);
}

Don't mind the code, it was just to try and see the speed.

Thank you for your help Smilie

EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.

Last edited by ganon551; 08-24-2015 at 01:23 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Maximum files in a Directory

Does Solaris impose limits on : - the maximum number of files a directory can have, - total file size in a directory If there is such limits, how can I can check for each? Thanks...:confused: (1 Reply)
Discussion started by: deaniyoer
1 Replies

2. UNIX for Dummies Questions & Answers

ls - maximum number of files

what is the maximum number ls can list down (6 Replies)
Discussion started by: karnan
6 Replies

3. Linux

maximum file & directories

Dear friends, How many maximum number of files and directories can be created in linux system. Thanks.. (2 Replies)
Discussion started by: karthigayan
2 Replies

4. UNIX for Dummies Questions & Answers

List directories and sub directories recursively excluding files

Hi, Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process. I wanted to disply using a unix command all the directories recursively excluding files. I tried 'ls -FR' but that display files as... (3 Replies)
Discussion started by: pointers
3 Replies

5. Shell Programming and Scripting

Sub directories containing maximum files

Hi All, I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain. find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10 But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Discussion started by: adurga
5 Replies

6. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

7. Shell Programming and Scripting

Rename the files in all the directories and sub-directories

Hi all, I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”. I need to... (1 Reply)
Discussion started by: AshwaniSharma09
1 Replies

8. Shell Programming and Scripting

Extracting files having maximum timestamp

Hi , I'm using Ksh 88 I've the following files in a directory with YearMonthDate (Ex:20130601) YDT:FILE1:20130601 YDT:FILE1:20130615 YDT:FILE2:20130601 YDT:FILE2:20130615 YDT:FILE3:20130601 YDT:FILE3:20130615 And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
Discussion started by: smile689
8 Replies

9. Shell Programming and Scripting

Archiving and moving files into directories, creating directories, etc.

how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex. so mydemo--2015-03-23-1500.tar.gz > database -... (5 Replies)
Discussion started by: wyclef
5 Replies
Digest::MD5(3pm)					 Perl Programmers Reference Guide					  Digest::MD5(3pm)

NAME
Digest::MD5 - Perl interface to the MD5 Algorithm SYNOPSIS
# Functional style use Digest::MD5 qw(md5 md5_hex md5_base64); $digest = md5($data); $digest = md5_hex($data); $digest = md5_base64($data); # OO style use Digest::MD5; $ctx = Digest::MD5->new; $ctx->add($data); $ctx->addfile(*FILE); $digest = $ctx->digest; $digest = $ctx->hexdigest; $digest = $ctx->b64digest; DESCRIPTION
The "Digest::MD5" module allows you to use the RSA Data Security Inc. MD5 Message Digest algorithm from within Perl programs. The algo- rithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. The "Digest::MD5" module provide a procedural interface for simple use, as well as an object oriented interface that can handle messages of arbitrary length and which can read files directly. A binary digest will be 16 bytes long. A hex digest will be 32 characters long. A base64 digest will be 22 characters long. FUNCTIONS
The following functions can be exported from the "Digest::MD5" module. No functions are exported by default. md5($data,...) This function will concatenate all arguments, calculate the MD5 digest of this "message", and return it in binary form. md5_hex($data,...) Same as md5(), but will return the digest in hexadecimal form. md5_base64($data,...) Same as md5(), but will return the digest as a base64 encoded string. The base64 encoded string returned is not padded to be a multiple of 4 bytes long. If you want interoperability with other base64 encoded md5 digests you might want to append the string "==" to the result. METHODS
The following methods are available: $md5 = Digest::MD5->new The constructor returns a new "Digest::MD5" object which encapsulate the state of the MD5 message-digest algorithm. You can add data to the object and finally ask for the digest. If called as an instance method (i.e. $md5->new) it will just reset the state the object to the state of a newly created object. No new object is created in this case. $md5->reset This is just an alias for $md5->new. $md5->add($data,...) The $data provided as argument are appended to the message we calculate the digest for. The return value is the $md5 object itself. $md5->addfile($io_handle) The $io_handle is read until EOF and the content is appended to the message we calculate the digest for. The return value is the $md5 object itself. In most cases you want to make sure that the $io_handle is set up to be in binmode(). $md5->digest Return the binary digest for the message. Note that the "digest" operation is effectively a destructive, read-once operation. Once it has been performed, the "Digest::MD5" object is automatically "reset" and can be used to calculate another digest value. $md5->hexdigest Same as $md5->digest, but will return the digest in hexadecimal form. $md5->b64digest Same as $md5->digest, but will return the digest as a base64 encoded string. The base64 encoded string returned is not padded to be a multiple of 4 bytes long. If you want interoperability with other base64 encoded md5 digests you might want to append the string "==" to the result. EXAMPLES
The simplest way to use this library is to import the md5_hex() function (or one of its cousins): use Digest::MD5 qw(md5_hex); print "Digest is ", md5_hex("foobarbaz"), " "; The above example would print out the message Digest is 6df23dc03f9b54cc38a0fc1483df6e21 provided that the implementation is working correctly. The same checksum can also be calculated in OO style: use Digest::MD5; $md5 = Digest::MD5->new; $md5->add('foo', 'bar'); $md5->add('baz'); $digest = $md5->hexdigest; print "Digest is $digest "; With OO style you can break the message arbitrary. This means that we are no longer limited to have space for the whole message in memory, i.e. we can handle messages of any size. This is useful when calculating checksum for files: use Digest::MD5; my $file = shift || "/etc/passwd"; open(FILE, $file) or die "Can't open '$file': $!"; binmode(FILE); $md5 = Digest::MD5->new; while (<FILE>) { $md5->add($_); } close(FILE); print $md5->b64digest, " $file "; Or we can use the builtin addfile method for more efficient reading of the file: use Digest::MD5; my $file = shift || "/etc/passwd"; open(FILE, $file) or die "Can't open '$file': $!"; binmode(FILE); print Digest::MD5->new->addfile(*FILE)->hexdigest, " $file "; SEE ALSO
Digest, Digest::MD2, Digest::SHA1, Digest::HMAC md5sum(1) RFC 1321 COPYRIGHT
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. Copyright 1998-2002 Gisle Aas. Copyright 1995-1996 Neil Winton. Copyright 1991-1992 RSA Data Security, Inc. The MD5 algorithm is defined in RFC 1321. The basic C code implementing the algorithm is derived from that in the RFC and is covered by the following copyright: o Copyright (C) 1991-2, RSA Data Security, Inc. Created 1991. All rights reserved. License to copy and use this software is granted provided that it is identified as the "RSA Data Security, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing this software or this function. License is also granted to make and use derivative works provided that such works are identified as "derived from the RSA Data Secu- rity, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing the derived work. RSA Data Security, Inc. makes no representations concerning either the merchantability of this software or the suitability of this software for any particular purpose. It is provided "as is" without express or implied warranty of any kind. These notices must be retained in any copies of any part of this documentation and/or software. This copyright does not prohibit distribution of any version of Perl containing this extension under the terms of the GNU or Artistic licenses. AUTHORS
The original MD5 interface was written by Neil Winton ("N.Winton@axion.bt.co.uk"). This release was made by Gisle Aas <gisle@ActiveState.com> perl v5.8.0 2002-06-01 Digest::MD5(3pm)
All times are GMT -4. The time now is 02:59 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy