Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Counting total files with different file types in each folder
Top Forums Shell Programming and Scripting Counting total files with different file types in each folder Post 302999719 by kchinnam on Monday 26th of June 2017 11:56:14 PM
Old 06-27-2017
Counting total files with different file types in each folder
Trying to count total files with different file types with thousands of files in each folder.
Since some files do not have extensions I have to use below criteria.

Count Total Files starting with --> "^ERROR"
Count Total Files starting with --> "^Runtime"
Count Everything else or files without any extension

sample input files in each sub-folder.
Code:
RuntimeProperties_296090758.xls
RuntimeProperties_296409844.xls
ERROR_261218287_296336046_20161213_101129
261218194_296090758_20161212_120448
RuntimeProperties_296413261.xls
ERROR_261218194_296090758_20161212_120448
261218287_296409844_20161213_120039
261218287_296336046_20161213_101129
ERROR_261218287_296409844_20161213_120040

Since I have to count this in a 12TB root folder with 6800 sub folders with thousands of files in each, this should not get into buffer overflow or out of memory or too many files situations. It should be faster.

I think either perl or awk can do this implicitly with the help of xargs!,, but not entirely sure how..

Code:
# I wish something like this can print counts for each sub-folder. 
for each targetDIR in $(6800 folders); do
     find $targetDIR -type f  | xargs -i awk -v file="{}"  -v td="$targetDIR" \
            'file ~ "./^ERROR" {CNT_ERROR += 1}; \
             file ~ "./^Runtime" {CNT_Runtime += 1}; \
             file !~ "./^ERROR|^./Runtime" {CNT_Others += 1}; \
             END {print td "," CNT_ERROR ","  CNT_Runtime "," CNT_Others}'
done

Then I can get overall counts myself.
Last edited by kchinnam; 06-27-2017 at 12:59 AM.. Reason: more verbose
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep running total/ final total across multiple files

Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)
Discussion started by: MrAd
5 Replies

2. Shell Programming and Scripting

list quantity of files by file types

I'm trying to create a simple file inventory for a series of huge directories containing e-records. What I'm after is a list of all directories and sub-directories with just the number of each type of file in that directory/sub-directory. For example output would look like: ... (6 Replies)
Discussion started by: dorcas
6 Replies

3. Shell Programming and Scripting

check how many files in folder or total files in folder

Hi all, I have copied all my files to one folder.and i want to check how many files (count) in the folder recently moved or total files in my folder? please send me the query asap. (3 Replies)
Discussion started by: durgaprasad
3 Replies

4. Shell Programming and Scripting

Getting the total file size for certain files per directory

Hi, I am trying to get the total file size for certain files per directory. I am using find /DirectoryPath -name '*.dta' -exec ls -l {} \; | awk '{ print $NF ": " $5 }' > /users/cergun/My\ Documents/dtafiles.txt but this lists all the files in the directories. I need the total... (9 Replies)
Discussion started by: cergun
9 Replies

5. UNIX for Dummies Questions & Answers

Counting total users

wow, back again :) I am trying to write a program that has many functions, however one of the functions will count the total number of users in the database. I want to just use wc -l passwd (i've made a copy of passwd to play with, no worries lol) but I am worried that if any of the... (4 Replies)
Discussion started by: SoVi3t
4 Replies

6. UNIX for Dummies Questions & Answers

I am trying to get the total size of a folder?

I am trying to get the total size of the folder using the below command but its not working. any ideas? du -bc <foldername>/|grep total|tr -s " "|cut -d" " -f1 the output i am getting is 78996 total but i just want it to be as 78996 please help (3 Replies)
Discussion started by: classic
3 Replies

7. Shell Programming and Scripting

Counting number of files that contain words stored in another file

Hi All, I have written a script on this but it does not do the requisite job. My requirement is this: 1. I have two kinds of files each with different extensions. One set of files are *.dat (6000 unique DAT files all in one directory) and another set *.dic files (6000 unique DIC files in... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

8. UNIX for Dummies Questions & Answers

[Solved] Touch 10% of the total files inside a folder

I'm trying to make this code below to work but I can't find the way to do the following: I want to make the script to touch only 10% of the total amount of files counted inside the given directory instead of all like it is now. I would greatly appreciate it if someone can give me a direction on... (9 Replies)
Discussion started by: regraphix
9 Replies

9. Shell Programming and Scripting

Total number of files in the folder should be listed

Hi All, When i give the ls -lrt to list out all files with total number of files , i get the output as ls -lrt total 72 -rw-r--r-- 1 hari staff 796 Jul 11 09:17 va.txt -rw-r--r-- 1 hari staff 169 Jul 13 00:20 a.log -rwxr-xr-x 1 hari staff 659 Aug... (9 Replies)
Discussion started by: Kalaihari
9 Replies

10. UNIX for Advanced & Expert Users

Searching for file types by count in specific folder in RHEL 6

So I'm trying to search for the top 10 or 15 items under a directory by file type. I want to run a command on a directory and get something like the following: Example of expected output.. .PDF: 100, .txt: 95, .word: 80.. What would be the best way of going about this? I've searched around... (2 Replies)
Discussion started by: shackle101
2 Replies

LEARN ABOUT ULTRIX

rmf

rmf(1mh)																  rmf(1mh)

Name
       rmf - remove folder

Syntax
       rmf [ +folder ] [ -help ] [ -[no]interactive ]

Description
       The  command  removes all of the messages within the current folder, and then removes the folder itself.  If there are any files within the
       folder which are not part of MH, they are not removed, and an error message is displayed.

       You can specify a folder other than the current folder by using the +folder argument.  If you do not specify a folder, and cannot find  the
       current folder, asks you whether you want to delete instead.

       If the current folder is removed, it makes current.

       Note that the command irreversibly deletes messages that do not have other links, so use it with caution.

       If  the folder being removed is a sub-folder, the parent folder becomes the new current folder, and tells you that this has happened.  This
       provides an easy mechanism for selecting a set of messages, operating on the list, then removing the list  and  returning  to  the  current
       folder from which the list was extracted.

       Using to delete a read-only folder deletes the private sequence and current message information from the file, without affecting the folder
       itself.	If you have sub-folders within a folder, you must delete all the sub-folders before you can delete the folder itself.

Options
       -help	 Prints a list of the valid options to this command.

       -interactive
       -nointeractive
		 Asks for confirmation before deleting a folder.  By default, deletes a folder and its messages without asking	for  confirmation.
		 If  you  specify  the	-interactive option, asks if you are sure before deleting the folder.  You are advised to use this option,
		 since when deletes a folder its contents are lost irretrievably.

Examples
       This example shows how asks for confirmation when the -interactive option is used:
       % rmf -interactive +test
       Remove folder "test"? y

Profile Components
       Path:   To determine the user's Mail directory

Files
       The user profile.

See Also
       rmm(1mh)

																	  rmf(1mh)
All times are GMT -4. The time now is 10:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy