Can awk do lookups to other files and process results
I know that 'brute-force' scripting could accomplish this with lots of cat/echo/cut/grep and more. But, because my real file has 800k records, and the matching files have 10-20k records, this is not time-possible or efficient.
I have input file:
(you can ignore the first line - just a help since a fixed record file)
(tail +2 file_in skips over this line during testing)
Begin by only reviewing records where position 40 is blank = still need to process.
Want to see those records that cannot be processed because (a) the data in columns 7-12 does not exist in the following file:
I know Joe does not match, so ideally I would like to put a "1" in position 39 telling me I failed the first test.
A second test (b) is to only process records that are "abc" based on lookup of columns 29-32 into the following file:
Linda & Matt should then have a "2" put in position 39.
So, my start would be
which would create an output file, but only records I want to even consider that are not yet marked as processed. So, yes I intend to start with 6 records and make a file of 5 records. I now need to add those two codes at position 39 when appropriate.
Okay. Use associative arrays. This gives you three files one.txt with a "1" two.txt three.txt which are intermediate and then bad.txt which is still just blank in col 39 & 40.
Last edited by jim mcnamara; 10-23-2008 at 05:24 PM..
After re-reading your post and Jim's comments I'm not sure if you prefer to generate multiple files (good - bad records) or an output like the one I posted.
I would prefer all data - good and bad records - stored to one file.
While reading through my 'sed & awk' book, the idea of arrays did jump out to me. I am going to have to sit and read through the examples to understand how they work.
I am using c shell and trying to compare 2 files using awk . But the below awk statement doesnt give any result. Pls. advise why am not getting the desired o/p with the corrected awk script.
Need to acheive this solution in awk using C shell.
awk 'FNR==NR{a++;next}
{for(i in a)
{if ( a=$0... (8 Replies)
Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk.
I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside.
For example:
a sample_1 200
a.b sample_2 10
a sample_3 10
a sample_1 10
a... (4 Replies)
Hi Experts,
I am facing one problem here which is one process always stuck in running state which causes the other similar process to sleep state . This causes my system in hanged state.
On doing cat /proc/<pid>wchan showing the "__init_begin" in the output.
Can you please help me here... (6 Replies)
Hi Experts,
I am facing one problem here which is one process always stuck in running state which causes the other similar process to sleep state . This causes my system in hanged state.
On doing cat /proc/<pid>wchan showing the "__init_begin" in the output.
Can you please help me here... (1 Reply)
Hi Experts,
I am facing one problem here which is one process always stuck in running state which causes the other similar process to sleep state . This causes my system in hanged state.
On doing cat /proc/<pid>wchan showing the "__init_begin" in the output.
Can you please help me here... (0 Replies)
Hi to all,
I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command
awk '/Code.*/' FILE-2014*
I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from
both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6.
Below is an... (5 Replies)
I am trying to parse two files and get data that does not match in one of the columns ( column 3 in my case )
Data for two files are as follows
A.txt
=====
abc 10 5 0 1 16
xyz 16 1 1 0 18
efg 30 8 0 2 40
ijk 22 2 0 1 25
B.txt
=====
abc... (6 Replies)
I am trying to match 4 colums (first_name,last_name,dob,ssn) between 2 files and when there is an exact match I need to write out these matches to a new file with a combination of fields from file1 and file2. I've managed to come up with a way to match these 2 files based on the columns (see below)... (7 Replies)
Just wondering if anyone else is using IBM's TWS on HP-UX 11.11i. Seeing some very strange name-lookup issues when it comes to using various utilities on the system. The same software works fine o0n AIX, Linux, Solaris, etc, but on HP-UX there is noticeable time lags in issuing commands - at the... (0 Replies)