Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Counting total files with different file types in each folder
Top Forums Shell Programming and Scripting Counting total files with different file types in each folder Post 302999719 by kchinnam on Monday 26th of June 2017 11:56:14 PM
Old 06-27-2017
Counting total files with different file types in each folder
Trying to count total files with different file types with thousands of files in each folder.
Since some files do not have extensions I have to use below criteria.

Count Total Files starting with --> "^ERROR"
Count Total Files starting with --> "^Runtime"
Count Everything else or files without any extension

sample input files in each sub-folder.
Code:
RuntimeProperties_296090758.xls
RuntimeProperties_296409844.xls
ERROR_261218287_296336046_20161213_101129
261218194_296090758_20161212_120448
RuntimeProperties_296413261.xls
ERROR_261218194_296090758_20161212_120448
261218287_296409844_20161213_120039
261218287_296336046_20161213_101129
ERROR_261218287_296409844_20161213_120040

Since I have to count this in a 12TB root folder with 6800 sub folders with thousands of files in each, this should not get into buffer overflow or out of memory or too many files situations. It should be faster.

I think either perl or awk can do this implicitly with the help of xargs!,, but not entirely sure how..

Code:
# I wish something like this can print counts for each sub-folder. 
for each targetDIR in $(6800 folders); do
     find $targetDIR -type f  | xargs -i awk -v file="{}"  -v td="$targetDIR" \
            'file ~ "./^ERROR" {CNT_ERROR += 1}; \
             file ~ "./^Runtime" {CNT_Runtime += 1}; \
             file !~ "./^ERROR|^./Runtime" {CNT_Others += 1}; \
             END {print td "," CNT_ERROR ","  CNT_Runtime "," CNT_Others}'
done

Then I can get overall counts myself.
Last edited by kchinnam; 06-27-2017 at 12:59 AM.. Reason: more verbose
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep running total/ final total across multiple files

Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Discussion started by: MrAd
5 Replies

2. Shell Programming and Scripting

list quantity of files by file types

I'm trying to create a simple file inventory for a series of huge directories containing e-records. What I'm after is a list of all directories and sub-directories with just the number of each type of file in that directory/sub-directory. For example output would look like: ... (6 Replies)
Discussion started by: dorcas
6 Replies

3. Shell Programming and Scripting

check how many files in folder or total files in folder

Hi all, I have copied all my files to one folder.and i want to check how many files (count) in the folder recently moved or total files in my folder? please send me the query asap. (3 Replies)
Discussion started by: durgaprasad
3 Replies

4. Shell Programming and Scripting

Getting the total file size for certain files per directory

Hi, I am trying to get the total file size for certain files per directory. I am using find /DirectoryPath -name '*.dta' -exec ls -l {} \; | awk '{ print $NF ": " $5 }' > /users/cergun/My\ Documents/dtafiles.txt but this lists all the files in the directories. I need the total... (9 Replies)
Discussion started by: cergun
9 Replies

5. UNIX for Dummies Questions & Answers

Counting total users

wow, back again :) I am trying to write a program that has many functions, however one of the functions will count the total number of users in the database. I want to just use wc -l passwd (i've made a copy of passwd to play with, no worries lol) but I am worried that if any of the... (4 Replies)
Discussion started by: SoVi3t
4 Replies

6. UNIX for Dummies Questions & Answers

I am trying to get the total size of a folder?

I am trying to get the total size of the folder using the below command but its not working. any ideas? du -bc <foldername>/|grep total|tr -s " "|cut -d" " -f1 the output i am getting is 78996 total but i just want it to be as 78996 please help (3 Replies)
Discussion started by: classic
3 Replies

7. Shell Programming and Scripting

Counting number of files that contain words stored in another file

Hi All, I have written a script on this but it does not do the requisite job. My requirement is this: 1. I have two kinds of files each with different extensions. One set of files are *.dat (6000 unique DAT files all in one directory) and another set *.dic files (6000 unique DIC files in... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

8. UNIX for Dummies Questions & Answers

[Solved] Touch 10% of the total files inside a folder

I'm trying to make this code below to work but I can't find the way to do the following: I want to make the script to touch only 10% of the total amount of files counted inside the given directory instead of all like it is now. I would greatly appreciate it if someone can give me a direction on... (9 Replies)
Discussion started by: regraphix
9 Replies

9. Shell Programming and Scripting

Total number of files in the folder should be listed

Hi All, When i give the ls -lrt to list out all files with total number of files , i get the output as ls -lrt total 72 -rw-r--r-- 1 hari staff 796 Jul 11 09:17 va.txt -rw-r--r-- 1 hari staff 169 Jul 13 00:20 a.log -rwxr-xr-x 1 hari staff 659 Aug... (9 Replies)
Discussion started by: Kalaihari
9 Replies

10. UNIX for Advanced & Expert Users

Searching for file types by count in specific folder in RHEL 6

So I'm trying to search for the top 10 or 15 items under a directory by file type. I want to run a command on a directory and get something like the following: Example of expected output.. .PDF: 100, .txt: 95, .word: 80.. What would be the best way of going about this? I've searched around... (2 Replies)
Discussion started by: shackle101
2 Replies

LEARN ABOUT LINUX

rmf

RMF(1)								     [nmh-1.5]								    RMF(1)

NAME

       rmf - remove an nmh folder

SYNOPSIS

       rmf [+folder] [-interactive | -nointeractive] [-version] [-help]

DESCRIPTION

       Rmf removes all of the messages (files) within the specified (or default) folder, and then removes the folder (directory) itself.

       If  there  are  any  files  within the folder which are not a part of nmh, they will not be removed, and an error will be produced.  If the
       folder is given explicitly or the -nointeractive option is given, then the folder will be removed  without  confirmation.   Otherwise,  the
       user  will be asked for confirmation.  If rmf can't find the current folder, for some reason, the folder to be removed defaults to `+inbox'
       (unless overridden by user's profile entry "Inbox") with confirmation.

       If the folder being removed is a subfolder, the parent folder will become the new current folder, and rmf will produce  a  message  telling
       the user this has happened.  This provides an easy mechanism for selecting a set of messages, operating on the list, then removing the list
       and returning to the current folder from which the list was extracted.

       If rmf s used on a read-only folder, it will delete all the (private) sequences (i.e., "atr-seq-folder" entries) for this folder from  your
       context without affecting the folder itself.

       Rmf irreversibly deletes messages that don't have other links, so use it with caution.

FILES

       $HOME/.mh_profile	  The user profile

PROFILE COMPONENTS

       Path:		    To determine the user's nmh directory
       Current-Folder:	    To find the default current folder
       Inbox:		    To find the default inbox

SEE ALSO

       rmm(1)

DEFAULTS

       `+folder' defaults to the current folder, usually with confirmation
       `-interactive' if +folder' not given, `-nointeractive' otherwise

CONTEXT

       Rmf  will  set the current folder to the parent folder if a subfolder is removed; or if the current folder is removed, it will make "inbox"
       current.  Otherwise, it doesn't change the current folder or message.

BUGS

       Although intuitively one would suspect that rmf works recursively, it does not.	Hence if you have a sub-folder within a folder,  in  order
       to rmf the parent, you must first rmf each of the children.

MH.6.8								   11 June 2012 							    RMF(1)
All times are GMT -4. The time now is 04:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy