Task ----- I need to search for the occurances of each Object_Name (from each row of File-1) in all the 5000 distinct files (names stored in File-2) and get the search results stored in some 3rd file with below row structure. So the total no of loop iterations would be 250,000,000.
Did you mention the format and number of records of the files listed in File-2?
What is the most powerful computer you have of the ones you mention?
You mention a performance issue. What have you tried so far?
I think the amount of data is reasonable to fit in one awk like this:
If the number of filenames is small enough, it'll run awk only twice, and grep only once, otherwise it will call grep as many times as necessary to open the 50,000 files and feed all its output through the one awk. If you're concerned about awk consuming too much memory, you can run grep | awk on individual files read from file-2 like
this will be less efficient, running awk and grep 50,000 times instead of 50,000/ARG_MAX times, but depending on the size of the files may not be significant.
Results when run on my own test data:
...where I'd created directories [a-z] with files [0-9] and put random object names from file-1 in them one per line. If you don't care what order you get the results in you can forget the sort.
Last edited by Corona688; 03-07-2011 at 01:41 PM..
I just checked your reply. I am at home and would give a try on your suggestion tomorrow only. I will post feedback immediately after I test the results.
Anyways, many thanks to you for such a detailed advice.
Did you mention the format and number of records of the files listed in File-2?
This especially is germaine. If file-2 consists of records with delimited fields and if the only matching to be done is field-wide (no partial matches), then this problem could be solved in a manner that's simpler and more efficient than Corona's approach (which handles generic text formats).
Regards,
Alister
---------- Post updated at 01:29 PM ---------- Previous update was at 01:08 PM ----------
Hey, Corona688:
A few observations:
Quote:
Originally Posted by Corona688
An FS=\| is necessary to properly split that file.
Quote:
Originally Posted by Corona688
Testing with the only version of grep that I have that supports -o (gnu grep 2.5.1 on a disused laptop that seldom sees any action), if multiple patterns match a single line (not knowing anything about the content of the files, it's a possibility), it does not print the filename before every match (only the first), even with -H. If that output format has changed, apologies for the false alarm.
Quote:
Originally Posted by Corona688
Actually, assuming the splitting on file-1 is done correctly and $2 is sent to the temp file, the grepping is being done on the object names, not the oids. The result of the grep and awk count will be filename|object name|count, but the desired output is filename|oid|count.
An FS=\| is necessary to properly split that file.
Good point.
Quote:
Testing with the only version of grep that I have that supports -o (gnu grep 2.5.1 on a disused laptop that seldom sees any action)
Nuts, I thought that was a portable option. Throw that whole script out, then.
Quote:
if multiple patterns match a single line (not knowing anything about the content of the files, it's a possibility), it does not print the filename before every match (only the first), even with -H.
I did note it expected one pattern per line.
Quote:
Actually, assuming the splitting on file-1 is done correctly and $2 is sent to the temp file, the grepping is being done on the object names
Argh. My script doesn't work, then.
Looking less and less like there's really going to be an efficient solution if you have to parse that thing the hard way line by individual line on systems and shells with no features.
---------- Post updated at 01:31 PM ---------- Previous update was at 12:49 PM ----------
OK, here's a brute-force version in awk:
Don' think it'll be quite as efficient as grep but ought to do. It ran in two seconds on 5,000 files cached, maybe 5-10 seconds uncached. I don't believe I used any GNU specific features either.
Put that in extract.awk and run with
You can change o[FILENAME "|" n[i]]++; to o[FILENAME "|" i]++; if I somehow got name vs ID backwards again.
---------- Post updated at 01:54 PM ---------- Previous update was at 01:31 PM ----------
Estimate that to work in 60 minutes on data similar to yours on a 'fast' machine. use 'xargs -n 10' to prevent it from eating all your memory. Maybe not so good. If there was some sort of pattern to the OIDs/names, that'd help a lot for finding them without having to brute-force check all 50,000 individually... Or if you knew for a fact you had or could get GNU grep, and didn't have more than one object per line, grep's hardwired performance is going to be way better than awk's scripted performance...
Last edited by Corona688; 03-07-2011 at 04:03 PM..
I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message
when I run the program. The program is not outputting to the csv file.
Error:
the file access permissions do not allow the specified action
cannot... (2 Replies)
I have around 300 files(*.rdf,*.fmb,*.pll,*.ctl,*.sh,*.sql,*.prog) which are of large size.
Around 8000 keywords(which will be in the file $keywordfile) needed to be searched inside those files.
If a keyword is found in a file..I have to insert the filename,extension,catagoery,keyword,occurrence... (8 Replies)
I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long"
I need to actually remove the last | symbol from this file.
sed -e 's/\|*$//' filename
is working fine for small files. But not working on this big... (13 Replies)
I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is:
while read line
do
emp_name=`echo $line`
grep -e... (7 Replies)
Hi everyone, newbie forum poster here. I'm an Oracle DBA and I require some guidance from the Unix gurus here about how to pinpoint where a problem is within a Solaris 9 system running on an 8 CPU Fujitsu server that acts as our Oracle database server. Our sysadmins are trying their best to... (13 Replies)
Hi,
I have a file which is around 193 gb in size. This file has tonnes of spaces and I need to sanitize it.
I tried to use awk script to split this file but it gave me an error like line to long...
As of now I am using a sed command to search replace the spaces; however its too slow for such a... (2 Replies)
Hello Gurus,
We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .
Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
I have a file that is 20 - 80+ MB in size that is a certain type of log file.
It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example:
The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)