Sponsored Content
Operating Systems Linux Debian Problem with maximum files and directories Post 302952938 by ganon551 on Monday 24th of August 2015 12:10:18 PM
Old 08-24-2015
Thank you for answering.

I should have post some code, that would have make things easier for sure.

I'm not trying to open 4096 files at the same time, I open the files, add a line, then close, and so on.

I decided to try with less directories, as in my previous example it would have make about 200 millions files. SO now I only have one directory with one hexadecimal letter. Then into each dir I have 4096 files, which is 65536 files at the end.

I also switched to perl as I thought it would be faster, but even with that the writing takes ages.

Here's some code :

Code:
#! /usr/bin/perl
use Digest::MD5 qw(md5_hex);
my $file = 'list';
open my $info, $file || die "Could not open $file: $!";
while( my $line = <$info>)  {   
    $line =~ s/\r|\n//g;
    my $md5 = md5_hex($line);
    my $add = substr $md5,0,1;
    my $add2 = substr $md5,1,3;
    my $add3 = substr $md5,4,3;
    $outfile = "md5hash__/".$add."/md5".$add2.".txt";
        open (FILE, ">> $outfile") || die "problem opening $outfile\n";
    print FILE $add3."\n".$line."\n";
    close(FILE);
}

Don't mind the code, it was just to try and see the speed.

Thank you for your help Smilie

EDIT : I should say, the file I'm opening for reading is about 25gB heavy. But that wasn't a problem when I only had 4096 files to write, so I thought it would be the same with 65536.

Last edited by ganon551; 08-24-2015 at 01:23 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Maximum files in a Directory

Does Solaris impose limits on : - the maximum number of files a directory can have, - total file size in a directory If there is such limits, how can I can check for each? Thanks...:confused: (1 Reply)
Discussion started by: deaniyoer
1 Replies

2. UNIX for Dummies Questions & Answers

ls - maximum number of files

what is the maximum number ls can list down (6 Replies)
Discussion started by: karnan
6 Replies

3. Linux

maximum file & directories

Dear friends, How many maximum number of files and directories can be created in linux system. Thanks.. (2 Replies)
Discussion started by: karthigayan
2 Replies

4. UNIX for Dummies Questions & Answers

List directories and sub directories recursively excluding files

Hi, Please help me, how to get all the direcotries, its sub directories and its sub directories recursively, need to exclude all the files in the process. I wanted to disply using a unix command all the directories recursively excluding files. I tried 'ls -FR' but that display files as... (3 Replies)
Discussion started by: pointers
3 Replies

5. Shell Programming and Scripting

Sub directories containing maximum files

Hi All, I have this command coded in C Shell to get the top ten sub directories in the order of number of files they contain. find $parent_dir -type d -exec filecount {} \; | sort -nr | head -10 But it does not seem to show any output. Can someone please help me out in correcting this... (5 Replies)
Discussion started by: adurga
5 Replies

6. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

7. Shell Programming and Scripting

Rename the files in all the directories and sub-directories

Hi all, I have more than 12000 files in 46 different directories and each directory has 2 sub-directories named “dat” or “gridded”. Dat sub-directories have files with extension “jpg.dat” and gridded sub-directories have files with extension “.jpg”. I need to... (1 Reply)
Discussion started by: AshwaniSharma09
1 Replies

8. Shell Programming and Scripting

Extracting files having maximum timestamp

Hi , I'm using Ksh 88 I've the following files in a directory with YearMonthDate (Ex:20130601) YDT:FILE1:20130601 YDT:FILE1:20130615 YDT:FILE2:20130601 YDT:FILE2:20130615 YDT:FILE3:20130601 YDT:FILE3:20130615 And i need the files having maximum timestamp , Means i need to display ... (8 Replies)
Discussion started by: smile689
8 Replies

9. Shell Programming and Scripting

Archiving and moving files into directories, creating directories, etc.

how can i move "dataName".sql.gz into a folder called 'database' and then move "$fileName".tar.gz * .htaccess into a folder called 'www' with the entire gzipped file being "$fileName".tar.gz? Is this doable or overly complex. so mydemo--2015-03-23-1500.tar.gz > database -... (5 Replies)
Discussion started by: wyclef
5 Replies
HFIND(1)						      General Commands Manual							  HFIND(1)

NAME
hfind - Lookup a hash value in a hash database SYNOPSIS
hfind [-i db_type ] [-f lookup_file ] [-eq] db_file [hashes] DESCRIPTION
hfind looks up hash values in a database using a binary search algorithm. This allows one to easily create a hash database and identify if a file is known or not. It works with the NIST National Software Reference Library (NSRL) and the output of 'md5sum'. Before the database can be used by 'hfind', an index file must be created with the '-i' option. This tool is needed for efficiency. Most text-based databases do not have fixed length entries and are sometimes not sorted. The hfind tool will create an index file that is sorted and has fixed-length entries. This allows for fast lookups using a binary search algorithm instead of a linear search such as 'grep'. ARGUMENTS
-i db_type Create an index file for the database. This step must be done before a lookup can be performed. The 'db_type' argument specifies the database type (i.e. nsrl-md5 or md5sum). See section below. -f lookup_file Specify the location of a file that contains one hash value per line. These hashes will be looked up in the database. -e Extended mode. Additional information besides just the name is printed. (Does not apply for all hash database types). -q Quick mode. Instead of displaying the corresponding information with the hash, just display 0 if the hash was not found and 1 if it was. If this flag is used, then only one hash can be given at a time. -V Display version db_file The location of the hash database file. [hashes] The hashes to lookup. If they are not supplied on the command line, STDIN is used. If index files exist for both SHA-1 and MD5 hashes, then both types of hashes can be given at runtime. INDEX FILE
hfind uses an index file to perform a binary search for a hash value. This is much faster than using 'grep', which will do a linear search. Before a hash database is used, a corresponding index file must be created. This is done with the '-i' option to hfind. The resulting index file will be named based on the database file name. The name will have the original name following by the hash type (sha1 or md5) followed by '.idx'. For example, creating an MD5 hash index of the NIST NSRL results in 'NSRLFile.txt-md5.idx' and the SHA-1 index results in 'NSRLFile.txt-sha1.idx'. The file has two columns. Each entry is sorted by the first column, which is the hash value. The second column has the byte offset of the corresponding entry in the original file. So, when a hash is found in the index, the offset is recorded and then 'hfind' seeks to the entry in the original database. The following input types are valid. For NSRL, 'nsrl-md5' and 'nsrl-sha1' can be used. The difference is which hash value the index is sorted by. The 'md5sum' value can also be used to sort and index "home made" databases. 'hfind' can take data in both common formats: MD5 (test.txt) = 76b1f4de1522c20b67acc132937cf82e and 76b1f4de1522c20b67acc132937cf82e test.txt EXAMPLES
To create an MD5 index file for NIST NSRL: # hfind -i nsrl-md5 /usr/local/hash/nsrl/NSRLFile.txt To lookup a value in the NSRL: # hfind /usr/local/hash/nsrl/NSRLFile.txt 76b1f4de1522c20b67acc132937cf82e 76b1f4de1522c20b67acc132937cf82e Hash Not Found You can even do both SHA-1 and MD5 if you want: # hfind -i nsrl-sha1 /usr/local/hash/nsrl/NSRLFile.txt # hfind /usr/local/hash/nsrl/NSRLFile.txt 76b1f4de1522c20b67acc132937cf82e 80001A80B3F1B80076B297CEE8805AAA04E1B5BA 76b1f4de1522c20b67acc132937cf82e Hash Not Found 80001A80B3F1B80076B297CEE8805AAA04E1B5BA thrdcore.cpp To make a database of critical binaries of a trusted system, use 'md5sum': # md5sum /bin/* /sbin/* /usr/bin/* /usr/bin/* /usr/local/bin/* /usr/local/sbin/* > system.md5 # hfind -i md5sum system.md5 To look entries up, the following will work: # hfind system.md5 76b1f4de1522c20b67acc132937cf82e 76b1f4de1522c20b67acc132937cf82e Hash Not Found or # md5sum -q /bin/* | hfind system.md5 928682269cd3edb1acdf9a7f7e606ff2 /bin/bash <...> or # md5sum -q /bin/* > bin.md5 # hfind -f bin.md5 system.md5 928682269cd3edb1acdf9a7f7e606ff2 /bin/bash <...> SEE ALSO
sorter(1) The NIST National Software Reference Library (NSRL) can be found at www.nsrl.nist.gov. LICENSE
Distributed under the Common Public License, found in the cpl1.0.txt file in the The Sleuth Kit licenses directory. AUTHOR
Brian Carrier <carrier at sleuthkit dot org> Send documentation updates to <doc-updates at sleuthkit dot org> HFIND(1)
All times are GMT -4. The time now is 08:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy