I should have post some code, that would have make things easier for sure.
I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.
I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.
I also switched to perl as I thought it would be faster, but even with that the writing takes ages.
Here's some code :
Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>) {
$line =~ s/\r|\n//g;
my $md5 = md5_hex($line);
my $add = substr $md5,0,1;
my $add2 = substr $md5,1,3;
my $add3 = substr $md5,4,3;
$outfile = "md5hash__/".$add."/md5".$add2.".txt";
open (FILE, ">> $outfile") || die "problem opening $outfile\n";
print FILE $add3."\n".$line."\n";
close(FILE);
}
Don't mind the code, it was just to try and see the speed.
Thank you for your help
EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.
Does Solaris impose limits on :
- the maximum number of files a directory can have,
- total file size in a directory
If there is such limits, how can I can check for each?
Thanks...:confused: (1 Reply)
Hi,
Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process.
I wanted to disply using a unix command all the directories recursively excluding files.
I tried 'ls -FR' but that display files as... (3 Replies)
Hi All,
I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain.
find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10
But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Can anyone come up with a unix command that lists
all the files, directories and sub-directories in the current directory
except a folder called log.?
Thank you in advance. (7 Replies)
Hi all,
I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”.
I need to... (1 Reply)
Hi ,
I'm using Ksh 88
I've the following files in a directory with YearMonthDate (Ex:20130601)
YDT:FILE1:20130601
YDT:FILE1:20130615
YDT:FILE2:20130601
YDT:FILE2:20130615
YDT:FILE3:20130601
YDT:FILE3:20130615
And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex.
so
mydemo--2015-03-23-1500.tar.gz
> database
-... (5 Replies)
Discussion started by: wyclef
5 Replies
LEARN ABOUT FREEBSD
ckdist
CKDIST(1) BSD General Commands Manual CKDIST(1)NAME
ckdist -- check software distributions
SYNOPSIS
ckdist [-airsx] [-d dir] [-n name] [-t type] file ...
DESCRIPTION
The ckdist utility reads ``checksum'' files (which are assumed to specify components of a software distribution) and verifies the integrity
of the distribution by validating the checksum of each component file. Both MD5 (128-bit ``message digest'') and .inf (32-bit CRC) checksum
formats are supported.
The file operands may refer to regular files or to directories. Regular files named md5, or which have an .md5 or an .inf extension, are
assumed to be of the implied type, otherwise format is determined from content. If a directory is specified, it is searched for appropri-
ately-named files only.
The options are as follows:
-a Report on all distribution components, not just those in respect of which errors are detected.
-i Ignore missing distribution components.
-r Search specified directories recursively.
-s Suppress complaints about inaccessible checksum files and directories.
-x Verify the existence of distribution components (and also check sizes, in the case of .inf files), but omit the more time-
consuming step of actually computing and comparing checksums.
-d dir Look for distribution components in the directory dir.
-n name Access distribution components using the filename name. When accessing .inf file components, append the appropriate exten-
sion to the filename.
-t type Assume that all specified checksum files are of the format type, and search directories only for files in this format (where
type is either md5 or inf).
EXIT STATUS
The ckdist utility exits with one of the following values:
0 No errors were detected.
1 Errors were found in a distribution.
2 Usage errors, inaccessible input files, or other system errors were encountered.
SEE ALSO cksum(1), md5(1)NOTES
Both BSD and DOS versions of ckdist are available.
BSD January 20, 1997 BSD
08-24-2015
