awk, associative array, compare files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk, associative array, compare files
# 8  
Old 10-19-2011
Network reply

no iam not able to procces in split second.I have started process one hour back its still in process.For missed_file1.txt genearation its self is in still process.
Code:
awk -F \; 'NR==FNR{a[$1];next} !($1 in a)' flie1.txt file2.txt > missed_file1.txt

can you please look at file3.txt, can we genreate three files using file3.txt.



Moderator's Comments:
Mod Comment Start to use code tags, thank you.

Last edited by zaxxon; 10-20-2011 at 04:42 AM.. Reason: code tags
# 9  
Old 10-20-2011
Network time

time taken to process the large for files for just to generate missed_file1.txt taken 9hours.so i need very efficient process using file3.txt can you please help if possible.
# 10  
Old 10-20-2011
You'll have to sort them first, then compare them line by line. Does the order of the output have to be the same as the order of the input?

Working on something.

---------- Post updated at 12:04 PM ---------- Previous update was at 10:29 AM ----------

Code:
$ cat missing.sh

#!/bin/sh

FA="data1"
FB="data2"

# Create temp files data1.1/data2.1 containing only first column.
# This lets us feed it into the 'comm' utility, which produces
# output we can quickly and easily process in awk.
awk -v FS=";" '{ print $1 >FILENAME ".1" }' ${FA} ${FB}
comm ${FA}.1 ${FB}.1 |
        awk -v FA="${FA}" -v FB="${FB}" -f missing.awk

# Delete temporary files
rm -f ${FA}.1 ${FB}.1


$ cat missing.awk
# Two tabs means third column, $1 is a token common to both files
/^\t\t\047/     {
                        getline AS<FA;  split(AS, A, ";");
                        getline BS<FB;  split(BS, B, ";");

                        for(N=2; N<=3; N++)
                        if(A[N] != B[N])
                        printf("for %s field%d differs %s:%s %s:%s\n",
                                A[1], N, FA, A[N], FB, B[N]) > "common.txt"
                }
# One tab means second column, $1 is only found in FB
/^\t\047/       {       if(getline <FB) print > "missed_" FB       }
# No tabs means first column, $1 is only found in FA
/^\047/         {       if(getline <FA) print > "missed_" FA       }

$ ./missing.sh

$ cat missed_data1

'393200103059';'TIM';'20110111'
'393200103061';'TIM';'20060206'
'393200103064';'OPI';'20110623'

$ cat missed_data2

'393200103056';'TIM';'20110111'
'393200103088';'TIM';'20060206'

$ cat common.txt

for '393200103052' field2 differs data1:'H3G' data2:'HKG'

$

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk Associative Array and/or Referring to Field by String (Nonconstant String Value)

I will start with an example of what I'm trying to do and then describe how I am approaching the issue. File PS028,005 Lexeme HRS # M # PhraseType 1(1:1) 7(7) PhraseLab 501 503 ClauseType ZYq0 PS028,005 Lexeme W # L> # BNH # M #... (17 Replies)
Discussion started by: jvoot
17 Replies

2. Shell Programming and Scripting

Associative array index question

I am trying to assign indexes to an associative array in a for loop but I have to use an eval command to make it work, this doesn't seem correct I don't have to do this with regular arrays For example, the following assignment fails without the eval command: #! /bin/bash read -d "\0" -a... (19 Replies)
Discussion started by: Riker1204
19 Replies

3. Shell Programming and Scripting

Using associative array for comparison

Hello together, i make something wrong... I want an array that contains information to associate it for further processing. Here is something from my bash... You will know, what I'm trying to do. I have to point out in advance, that the variable $SYSOS is changing and not as static as in my... (2 Replies)
Discussion started by: Decstasy
2 Replies

4. Shell Programming and Scripting

Awk: Dealing with whitespace in associative array indicies

Is there a reliable way to deal with whitespace in array indicies? I am trying to annotate fails in a database using a table of known fails. In a begin block I have code like this: # Read in Known Fail List getline < "'"$failListFile"'"; getline < "'"$failListFile"'"; getline <... (6 Replies)
Discussion started by: Michael Stora
6 Replies

5. Shell Programming and Scripting

Morse Code with Associative Array

Continuing my quest to learn BASH, Bourne, Awk, Grep, etc. on my own through the use of a few books. I've come to an exercise that has me absolutely stumped. The specifics: 1. Using ONLY BASH scripting commands (not sed, awk, etc.), write a script to convert a string on the command line to... (22 Replies)
Discussion started by: ksmarine1980
22 Replies

6. Shell Programming and Scripting

Bash 3d associative array with bash3 AND multiple files

Hello again guru’s (big apologies for wall of text) Still working on that DNS updater for my production team and while there is a ton of hit in searches i can't seem to find the answer to this. Context: We have apps that switch from let’s say host1 to host2. REAL basic DNS clustering... (5 Replies)
Discussion started by: maverick72
5 Replies

7. Shell Programming and Scripting

Associative array

I have an associative array named table declare -A table table="fruit" table="veggie" table="GT" table="eminem" Now say I have a variable returning the value highway How do I find corresponding value GT ?? (this value that I find (GT in this case) is supposed to be the name of a mysql... (1 Reply)
Discussion started by: leghorn
1 Replies

8. Shell Programming and Scripting

Help needed on Associative array in awk

Hi All, I got stuck up with shell script where i use awk. The scenario which i am working on is as below. I have a file text.txt with contents COL1 COL2 COL3 COL4 1 A 500 400 1 B 500 400 1 A 500 200 2 A 290 300 2 B 290 280 3 C 100 100 I could able to sum col 3 and col4 based on... (3 Replies)
Discussion started by: imsularif
3 Replies

9. Shell Programming and Scripting

Problem with lookup values on AWK associative array

I'm at wits end with this issue and my troubleshooting leads me to believe it is a problem with the file formatting of the array referenced by my script: awk -F, '{if (NR==FNR) {a=$4","$3","$2}\ else {print a "," $0}}' WBTSassignments1.txt RNCalarms.tmp On the WBTSassignments1.txt file... (2 Replies)
Discussion started by: JasonHamm
2 Replies

10. Shell Programming and Scripting

Associative Array

Hi, I am trying to make an associative array to use in a popup_menu on a website. Here is what i have: foreach $entr ( @entries ) { $temp_uid = $entr->get_value(uid); $temp_naam = $entr->get_value(sn); $s++; } This is the popup_menu i want to use it in. popup_menu(-name=>'modcon',... (4 Replies)
Discussion started by: tine
4 Replies
Login or Register to Ask a Question