Performance of calculating total number of matching records in multiple files
Hello Friends,
I've been trying to calculate total number of a certain match in multiple data records files (DRs).
Let say I have a daily created folders for each day since the beginning of july like the following
Each folder has tousands of files such as :
And unfortunately each file has hundreds or several tousands rows (data records) whose FS are commas and in which they have a whole soap request as a field. For example the following is one charging request record (I have shortened it)
I need to calculate total number of spesific matching i.e. $1 = charging, $9 = Subscriber.. and I calculated it but under just one folder:
Here my question:
As there are hunder tousand files under a few directories how should I calculate the total number of matching fastest way by executing only a one-liner command or script?
Should I first find the files with a find command in a FOR loop and trigger nawk afterwards like the following?
I would appreciate your suggestions. I checked but could not find a spesific answer on our forum or some others.
Your suggested method would start a new nawk on every file and probably be very slow.
Using + terminator instead of ; with find's -exec groups many files together. It's still possible that it ends up being several invocations of nawk because of argument limits. Then we need to somehow keep track of the sum and pass the value around. One solution is to have find just cat all the files and pipe it to nawk.
In two previous posts (here) and (here), I received help from forum members comparing multiple fields across two files and selectively printing portions of each as output based upon would-be matches using awk. I had been fairly comfortable populating awk arrays with fields and using awk's special... (3 Replies)
Hello people,
On HP-UX B.11.11 U 9000/800
How can I have in aprox. the total number of files in a specific FS?
Is the number of used inodes a rough estimation of my total number of files?
Server1 /Data:df -i .
/Data (/dev/vg_Data/lvol1 ) : 18292960 total i-nodes
15800945 free... (3 Replies)
Hi Experts,
In a particular dir, I have many files *AJAY*.
How can I get total size of all such files.
I tried du -hs *AJAY* but it gave me individual size of all files.
All I require is summation of all.
Thanks,
Ajay (4 Replies)
Hello Friends,
I know you all are busy and inteligent too...
I am stuck with one small issue if you can help me then it will be really great.
My problem is I am having some files i.e.
Input.txt1
Input.txt2
Input.txt3
Now my task is I need to check the total number of rows in... (4 Replies)
Good Morning/Afternoon All,
I am having some trouble creating a variable called "total" to display the sum of the values in a specific field, $6 for example.
The data I am working on is in the following form:
John Doe:(555) 555-5555:1:2:3
Jane Doe:(544) 444-5556:4:5:6
Moe Doe:(654)... (2 Replies)
Hi,
I have CSV file which looks like below, i want to calulate number of records for each brand say SOLO_UNBEATABLE E and SOLO_UNBEATABLE F combined and record count is say 20 . i want to calculate for each brand, and here only first record will have all data and rest of record for the brand... (2 Replies)
Hi All,
I have 2 files (file1 & file2).
File1 and File2 have m and n columns respectively
I have to compare value in column1 of file1 with file2 and find line(s) from file2 matching column1 value.
The value can be in any column in the matching lines of file2.
The output should be... (10 Replies)
Good afternoon! Im new at scripting and Im trying to write a script to
calculate total space, total used space and total free space in filesystem names matching a keyword (in this one we will use keyword virginia). Please dont be mean or harsh, like I said Im new and trying my best. Scripting... (4 Replies)
Ok, another fun hiccup in my UNIX learning curve. I am trying to count the number of occurrences of an IP address across multiple files named example.hits. I can extract the number of occurrences from the files individually but when you use grep -c with multiple files you get the output similar to... (5 Replies)