Sponsored Content
Operating Systems Linux Debian Problem with maximum files and directories Post 302952938 by ganon551 on Monday 24th of August 2015 12:10:18 PM
Old 08-24-2015
Thank you for answering.

I should have post some code, that would have make things easier for sure.

I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.

I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.

I also switched to perl as I thought it would be faster, but even with that the writing takes ages.

Here's some code :

Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>)  {   
    $line =~ s/\r|\n//g;
    my $md5 = md5_hex($line);
    my $add = substr $md5,0,1;
    my $add2 = substr $md5,1,3;
    my $add3 = substr $md5,4,3;
    $outfile = "md5hash__/".$add."/md5".$add2.".txt";
        open (FILE, ">> $outfile") || die "problem opening $outfile\n";
    print FILE $add3."\n".$line."\n";
    close(FILE);
}

Don't mind the code, it was just to try and see the speed.

Thank you for your help Smilie

EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.

Last edited by ganon551; 08-24-2015 at 01:23 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Maximum files in a Directory

Does Solaris impose limits on : - the maximum number of files a directory can have, - total file size in a directory If there is such limits, how can I can check for each? Thanks...:confused: (1 Reply)
Discussion started by: deaniyoer
1 Replies

2. UNIX for Dummies Questions & Answers

ls - maximum number of files

what is the maximum number ls can list down (6 Replies)
Discussion started by: karnan
6 Replies

3. Linux

maximum file & directories

Dear friends, How many maximum number of files and directories can be created in linux system. Thanks.. (2 Replies)
Discussion started by: karthigayan
2 Replies

4. UNIX for Dummies Questions & Answers

List directories and sub directories recursively excluding files

Hi, Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process. I wanted to disply using a unix command all the directories recursively excluding files. I tried 'ls -FR' but that display files as... (3 Replies)
Discussion started by: pointers
3 Replies

5. Shell Programming and Scripting

Sub directories containing maximum files

Hi All, I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain. find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10 But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Discussion started by: adurga
5 Replies

6. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

7. Shell Programming and Scripting

Rename the files in all the directories and sub-directories

Hi all, I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”. I need to... (1 Reply)
Discussion started by: AshwaniSharma09
1 Replies

8. Shell Programming and Scripting

Extracting files having maximum timestamp

Hi , I'm using Ksh 88 I've the following files in a directory with YearMonthDate (Ex:20130601) YDT:FILE1:20130601 YDT:FILE1:20130615 YDT:FILE2:20130601 YDT:FILE2:20130615 YDT:FILE3:20130601 YDT:FILE3:20130615 And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
Discussion started by: smile689
8 Replies

9. Shell Programming and Scripting

Archiving and moving files into directories, creating directories, etc.

how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex. so mydemo--2015-03-23-1500.tar.gz > database -... (5 Replies)
Discussion started by: wyclef
5 Replies
Digest::MD5(3pm)					 Perl Programmers Reference Guide					  Digest::MD5(3pm)

NAME
Digest::MD5 - Perl interface to the MD5 Algorithm SYNOPSIS
# Functional style use Digest::MD5 qw(md5 md5_hex md5_base64); $digest = md5($data); $digest = md5_hex($data); $digest = md5_base64($data); # OO style use Digest::MD5; $ctx = Digest::MD5->new; $ctx->add($data); $ctx->addfile(*FILE); $digest = $ctx->digest; $digest = $ctx->hexdigest; $digest = $ctx->b64digest; DESCRIPTION
The "Digest::MD5" module allows you to use the RSA Data Security Inc. MD5 Message Digest algorithm from within Perl programs. The algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. Note that the MD5 algorithm is not as strong as it used to be. It has since 2005 been easy to generate different messages that produce the same MD5 digest. It still seems hard to generate messages that produce a given digest, but it is probably wise to move to stronger algorithms for applications that depend on the digest to uniquely identify a message. The "Digest::MD5" module provide a procedural interface for simple use, as well as an object oriented interface that can handle messages of arbitrary length and which can read files directly. FUNCTIONS
The following functions are provided by the "Digest::MD5" module. None of these functions are exported by default. md5($data,...) This function will concatenate all arguments, calculate the MD5 digest of this "message", and return it in binary form. The returned string will be 16 bytes long. The result of md5("a", "b", "c") will be exactly the same as the result of md5("abc"). md5_hex($data,...) Same as md5(), but will return the digest in hexadecimal form. The length of the returned string will be 32 and it will only contain characters from this set: '0'..'9' and 'a'..'f'. md5_base64($data,...) Same as md5(), but will return the digest as a base64 encoded string. The length of the returned string will be 22 and it will only contain characters from this set: 'A'..'Z', 'a'..'z', '0'..'9', '+' and '/'. Note that the base64 encoded string returned is not padded to be a multiple of 4 bytes long. If you want interoperability with other base64 encoded md5 digests you might want to append the redundant string "==" to the result. METHODS
The object oriented interface to "Digest::MD5" is described in this section. After a "Digest::MD5" object has been created, you will add data to it and finally ask for the digest in a suitable format. A single object can be used to calculate multiple digests. The following methods are provided: $md5 = Digest::MD5->new The constructor returns a new "Digest::MD5" object which encapsulate the state of the MD5 message-digest algorithm. If called as an instance method (i.e. $md5->new) it will just reset the state the object to the state of a newly created object. No new object is created in this case. $md5->reset This is just an alias for $md5->new. $md5->clone This a copy of the $md5 object. It is useful when you do not want to destroy the digests state, but need an intermediate value of the digest, e.g. when calculating digests iteratively on a continuous data stream. Example: my $md5 = Digest::MD5->new; while (<>) { $md5->add($_); print "Line $.: ", $md5->clone->hexdigest, " "; } $md5->add($data,...) The $data provided as argument are appended to the message we calculate the digest for. The return value is the $md5 object itself. All these lines will have the same effect on the state of the $md5 object: $md5->add("a"); $md5->add("b"); $md5->add("c"); $md5->add("a")->add("b")->add("c"); $md5->add("a", "b", "c"); $md5->add("abc"); $md5->addfile($io_handle) The $io_handle will be read until EOF and its content appended to the message we calculate the digest for. The return value is the $md5 object itself. The addfile() method will croak() if it fails reading data for some reason. If it croaks it is unpredictable what the state of the $md5 object will be in. The addfile() method might have been able to read the file partially before it failed. It is probably wise to discard or reset the $md5 object if this occurs. In most cases you want to make sure that the $io_handle is in "binmode" before you pass it as argument to the addfile() method. $md5->add_bits($data, $nbits) $md5->add_bits($bitstring) Since the MD5 algorithm is byte oriented you might only add bits as multiples of 8, so you probably want to just use add() instead. The add_bits() method is provided for compatibility with other digest implementations. See Digest for description of the arguments that add_bits() take. $md5->digest Return the binary digest for the message. The returned string will be 16 bytes long. Note that the "digest" operation is effectively a destructive, read-once operation. Once it has been performed, the "Digest::MD5" object is automatically "reset" and can be used to calculate another digest value. Call $md5->clone->digest if you want to calculate the digest without resetting the digest state. $md5->hexdigest Same as $md5->digest, but will return the digest in hexadecimal form. The length of the returned string will be 32 and it will only contain characters from this set: '0'..'9' and 'a'..'f'. $md5->b64digest Same as $md5->digest, but will return the digest as a base64 encoded string. The length of the returned string will be 22 and it will only contain characters from this set: 'A'..'Z', 'a'..'z', '0'..'9', '+' and '/'. The base64 encoded string returned is not padded to be a multiple of 4 bytes long. If you want interoperability with other base64 encoded md5 digests you might want to append the string "==" to the result. EXAMPLES
The simplest way to use this library is to import the md5_hex() function (or one of its cousins): use Digest::MD5 qw(md5_hex); print "Digest is ", md5_hex("foobarbaz"), " "; The above example would print out the message: Digest is 6df23dc03f9b54cc38a0fc1483df6e21 The same checksum can also be calculated in OO style: use Digest::MD5; $md5 = Digest::MD5->new; $md5->add('foo', 'bar'); $md5->add('baz'); $digest = $md5->hexdigest; print "Digest is $digest "; With OO style you can break the message arbitrary. This means that we are no longer limited to have space for the whole message in memory, i.e. we can handle messages of any size. This is useful when calculating checksum for files: use Digest::MD5; my $file = shift || "/etc/passwd"; open(FILE, $file) or die "Can't open '$file': $!"; binmode(FILE); $md5 = Digest::MD5->new; while (<FILE>) { $md5->add($_); } close(FILE); print $md5->b64digest, " $file "; Or we can use the addfile method for more efficient reading of the file: use Digest::MD5; my $file = shift || "/etc/passwd"; open(FILE, $file) or die "Can't open '$file': $!"; binmode(FILE); print Digest::MD5->new->addfile(*FILE)->hexdigest, " $file "; Perl 5.8 support Unicode characters in strings. Since the MD5 algorithm is only defined for strings of bytes, it can not be used on strings that contains chars with ordinal number above 255. The MD5 functions and methods will croak if you try to feed them such input data: use Digest::MD5 qw(md5_hex); my $str = "abcx{300}"; print md5_hex($str), " "; # croaks # Wide character in subroutine entry What you can do is calculate the MD5 checksum of the UTF-8 representation of such strings. This is achieved by filtering the string through encode_utf8() function: use Digest::MD5 qw(md5_hex); use Encode qw(encode_utf8); my $str = "abcx{300}"; print md5_hex(encode_utf8($str)), " "; # 8c2d46911f3f5a326455f0ed7a8ed3b3 SEE ALSO
Digest, Digest::MD2, Digest::SHA, Digest::HMAC md5sum(1) RFC 1321 http://en.wikipedia.org/wiki/MD5 The paper "How to Break MD5 and Other Hash Functions" by Xiaoyun Wang and Hongbo Yu. COPYRIGHT
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. Copyright 1998-2003 Gisle Aas. Copyright 1995-1996 Neil Winton. Copyright 1991-1992 RSA Data Security, Inc. The MD5 algorithm is defined in RFC 1321. This implementation is derived from the reference C code in RFC 1321 which is covered by the following copyright statement: o Copyright (C) 1991-2, RSA Data Security, Inc. Created 1991. All rights reserved. License to copy and use this software is granted provided that it is identified as the "RSA Data Security, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing this software or this function. License is also granted to make and use derivative works provided that such works are identified as "derived from the RSA Data Security, Inc. MD5 Message-Digest Algorithm" in all material mentioning or referencing the derived work. RSA Data Security, Inc. makes no representations concerning either the merchantability of this software or the suitability of this software for any particular purpose. It is provided "as is" without express or implied warranty of any kind. These notices must be retained in any copies of any part of this documentation and/or software. This copyright does not prohibit distribution of any version of Perl containing this extension under the terms of the GNU or Artistic licenses. AUTHORS
The original "MD5" interface was written by Neil Winton ("N.Winton@axion.bt.co.uk"). The "Digest::MD5" module is written by Gisle Aas <gisle@ActiveState.com>. perl v5.16.2 2012-10-25 Digest::MD5(3pm)
All times are GMT -4. The time now is 10:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy