I should have post some code, that would have make things easier for sure.
I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.
I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.
I also switched to perl as I thought it would be faster, but even with that the writing takes ages.
Here's some code :
Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>) {
$line =~ s/\r|\n//g;
my $md5 = md5_hex($line);
my $add = substr $md5,0,1;
my $add2 = substr $md5,1,3;
my $add3 = substr $md5,4,3;
$outfile = "md5hash__/".$add."/md5".$add2.".txt";
open (FILE, ">> $outfile") || die "problem opening $outfile\n";
print FILE $add3."\n".$line."\n";
close(FILE);
}
Don't mind the code, it was just to try and see the speed.
Thank you for your help
EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.
Does Solaris impose limits on :
- the maximum number of files a directory can have,
- total file size in a directory
If there is such limits, how can I can check for each?
Thanks...:confused: (1 Reply)
Hi,
Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process.
I wanted to disply using a unix command all the directories recursively excluding files.
I tried 'ls -FR' but that display files as... (3 Replies)
Hi All,
I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain.
find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10
But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Can anyone come up with a unix command that lists
all the files, directories and sub-directories in the current directory
except a folder called log.?
Thank you in advance. (7 Replies)
Hi all,
I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”.
I need to... (1 Reply)
Hi ,
I'm using Ksh 88
I've the following files in a directory with YearMonthDate (Ex:20130601)
YDT:FILE1:20130601
YDT:FILE1:20130615
YDT:FILE2:20130601
YDT:FILE2:20130615
YDT:FILE3:20130601
YDT:FILE3:20130615
And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex.
so
mydemo--2015-03-23-1500.tar.gz
> database
-... (5 Replies)
Discussion started by: wyclef
5 Replies
LEARN ABOUT OSX
md5
MD5(1) BSD General Commands Manual MD5(1)NAME
md5 -- calculate a message-digest fingerprint (checksum) for a file
SYNOPSIS
md5 [-pqrtx] [-s string] [file ...]
DESCRIPTION
The md5 utility takes as input a message of arbitrary length and produces as output a ``fingerprint'' or ``message digest'' of the input. It
is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having
a given prespecified target message digest. The MD5 algorithm is intended for digital signature applications, where a large file must be
``compressed'' in a secure manner before being encrypted with a private (secret) key under a public-key cryptosystem such as RSA.
MD5's designer Ron Rivest has stated "md5 and sha1 are both clearly broken (in terms of collision-resistance)". So MD5 should be avoided
when creating new protocols, or implementing protocols with better options. SHA256 and SHA512 are better options as they have been more
resilient to attacks (as of 2009).
The following options may be used in any combination and must precede any files named on the command line. The hexadecimal checksum of each
file listed on the command line is printed after the options are processed.
-s string
Print a checksum of the given string.
-p Echo stdin to stdout and append the checksum to stdout.
-q Quiet mode - only the checksum is printed out. Overrides the -r option.
-r Reverses the format of the output. This helps with visual diffs. Does nothing when combined with the -ptx options.
-t Run a built-in time trial.
-x Run a built-in test script.
EXIT STATUS
The md5 utility exits 0 on success, and 1 if at least one of the input files could not be read.
SEE ALSO cksum(1), md5(3), ripemd(3), sha(3), CC_SHA256_Init(3)
Vlastimil Klima, Finding MD5 Collisions - a Toy For a Notebook, Cryptology ePrint Archive: Report 2005/075.
ACKNOWLEDGMENTS
This program is placed in the public domain for free general use by RSA Data Security.
BSD June 6, 2004 BSD
08-24-2015
