Sponsored Content
Operating Systems Linux Debian Problem with maximum files and directories Post 302952938 by ganon551 on Monday 24th of August 2015 12:10:18 PM
Old 08-24-2015
Thank you for answering.

I should have post some code, that would have make things easier for sure.

I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.

I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.

I also switched to perl as I thought it would be faster, but even with that the writing takes ages.

Here's some code :

Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>)  {   
    $line =~ s/\r|\n//g;
    my $md5 = md5_hex($line);
    my $add = substr $md5,0,1;
    my $add2 = substr $md5,1,3;
    my $add3 = substr $md5,4,3;
    $outfile = "md5hash__/".$add."/md5".$add2.".txt";
        open (FILE, ">> $outfile") || die "problem opening $outfile\n";
    print FILE $add3."\n".$line."\n";
    close(FILE);
}

Don't mind the code, it was just to try and see the speed.

Thank you for your help Smilie

EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.

Last edited by ganon551; 08-24-2015 at 01:23 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Maximum files in a Directory

Does Solaris impose limits on : - the maximum number of files a directory can have, - total file size in a directory If there is such limits, how can I can check for each? Thanks...:confused: (1 Reply)
Discussion started by: deaniyoer
1 Replies

2. UNIX for Dummies Questions & Answers

ls - maximum number of files

what is the maximum number ls can list down (6 Replies)
Discussion started by: karnan
6 Replies

3. Linux

maximum file & directories

Dear friends, How many maximum number of files and directories can be created in linux system. Thanks.. (2 Replies)
Discussion started by: karthigayan
2 Replies

4. UNIX for Dummies Questions & Answers

List directories and sub directories recursively excluding files

Hi, Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process. I wanted to disply using a unix command all the directories recursively excluding files. I tried 'ls -FR' but that display files as... (3 Replies)
Discussion started by: pointers
3 Replies

5. Shell Programming and Scripting

Sub directories containing maximum files

Hi All, I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain. find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10 But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Discussion started by: adurga
5 Replies

6. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

7. Shell Programming and Scripting

Rename the files in all the directories and sub-directories

Hi all, I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”. I need to... (1 Reply)
Discussion started by: AshwaniSharma09
1 Replies

8. Shell Programming and Scripting

Extracting files having maximum timestamp

Hi , I'm using Ksh 88 I've the following files in a directory with YearMonthDate (Ex:20130601) YDT:FILE1:20130601 YDT:FILE1:20130615 YDT:FILE2:20130601 YDT:FILE2:20130615 YDT:FILE3:20130601 YDT:FILE3:20130615 And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
Discussion started by: smile689
8 Replies

9. Shell Programming and Scripting

Archiving and moving files into directories, creating directories, etc.

how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex. so mydemo--2015-03-23-1500.tar.gz > database -... (5 Replies)
Discussion started by: wyclef
5 Replies
md5(n)							   MD5 Message-Digest Algorithm 						    md5(n)

__________________________________________________________________________________________________________________________________________________

NAME
md5 - MD5 Message-Digest Algorithm SYNOPSIS
package require Tcl 8.2 package require md5 ?2.0.7? ::md5::md5 ?-hex? [ -channel channel | -file filename | string ] ::md5::hmac ?-hex? -key key [ -channel channel | -file filename | string ] ::md5::MD5Init ::md5::MD5Update token data ::md5::MD5Final token ::md5::HMACInit key ::md5::HMACUpdate token data ::md5::HMACFinal token _________________________________________________________________ DESCRIPTION
This package is an implementation in Tcl of the MD5 message-digest algorithm as described in RFC 1321 (1). This algorithm takes an arbi- trary quantity of data and generates a 128-bit message digest from the input. The MD5 algorithm is related to the MD4 algorithm (2) but has been strengthened against certain types of potential attack. MD5 should be used in preference to MD4 for new applications. If you have critcl and have built the tcllibc package then the implementation of the hashing function will be performed by compiled code. Alternatively if you have either cryptkit or Trf then either of these can be used to accelerate the digest computation. If no suitable com- piled package is available then the pure-Tcl implementation wil be used. The programming interface remains the same in all cases. Note the previous version of this package always returned a hex encoded string. This has been changed to simplify the programming interface and to make this version more compatible with other implementations. To obtain the previous usage, either explicitly specify package ver- sion 1 or use the -hex option to the md5 command. COMMANDS
::md5::md5 ?-hex? [ -channel channel | -file filename | string ] Calculate the MD5 digest of the data given in string. This is returned as a binary string by default. Giving the -hex option will return a hexadecimal encoded version of the digest. The data to be hashed can be specified either as a string argument to the md5 command, or as a filename or a pre-opened channel. If the -filename argument is given then the file is opened, the data read and hashed and the file is closed. If the -channel argument is given then data is read from the channel until the end of file. The channel is not closed. Only one of -file, -channel or string should be given. ::md5::hmac ?-hex? -key key [ -channel channel | -file filename | string ] Calculate an Hashed Message Authentication digest (HMAC) using the MD5 digest algorithm. HMACs are described in RFC 2104 (3) and provide an MD5 digest that includes a key. All options other than -key are as for the ::md5::md5 command. PROGRAMMING INTERFACE
For the programmer, the MD5 hash can be viewed as a bucket into which one pours data. When you have finished, you extract a value that is derived from the data that was poured into the bucket. The programming interface to the MD5 hash operates on a token (equivalent to the bucket). You call MD5Init to obtain a token and then call MD5Update as many times as required to add data to the hash. To release any resources and obtain the hash value, you then call MD5Final. An equivalent set of functions gives you a keyed digest (HMAC). ::md5::MD5Init Begins a new MD5 hash. Returns a token ID that must be used for the remaining functions. ::md5::MD5Update token data Add data to the hash identified by token. Calling MD5Update $token "abcd" is equivalent to calling MD5Update $token "ab" followed by MD5Update $token "cb". See EXAMPLES. ::md5::MD5Final token Returns the hash value and releases any resources held by this token. Once this command completes the token will be invalid. The result is a binary string of 16 bytes representing the 128 bit MD5 digest value. ::md5::HMACInit key This is equivalent to the ::md5::MD5Init command except that it requires the key that will be included in the HMAC. ::md5::HMACUpdate token data ::md5::HMACFinal token These commands are identical to the MD5 equivalent commands. EXAMPLES
% md5::md5 -hex "Tcl does MD5" 8AAC1EE01E20BB347104FABB90310433 % md5::hmac -hex -key Sekret "Tcl does MD5" 35BBA244FD56D3EDF5F3C47474DACB5D % set tok [md5::MD5Init] ::md5::1 % md5::MD5Update $tok "Tcl " % md5::MD5Update $tok "does " % md5::MD5Update $tok "MD5" % md5::Hex [md5::MD5Final $tok] 8AAC1EE01E20BB347104FABB90310433 REFERENCES
[1] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, MIT and RSA Data Security, Inc, April 1992. (http://www.rfc-edi- tor.org/rfc/rfc1321.txt) [2] Rivest, R., "The MD4 Message Digest Algorithm", RFC 1320, MIT, April 1992. (http://www.rfc-editor.org/rfc/rfc1320.txt) [3] Krawczyk, H., Bellare, M. and Canetti, R. "HMAC: Keyed-Hashing for Message Authentication", RFC 2104, February 1997. (http://www.rfc-editor.org/rfc/rfc2104.txt) BUGS, IDEAS, FEEDBACK This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category md5 of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for either package and/or documentation. SEE ALSO
md4, sha1 KEYWORDS
hashing, md5, message-digest, rfc 1320, rfc 1321, rfc 2104, security COPYRIGHT
Copyright (c) 2003, Pat Thoyts <patthoyts@users.sourceforge.net> md5 2.0.7 md5(n)
All times are GMT -4. The time now is 04:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy