Sponsored Content
Top Forums Shell Programming and Scripting Problem in comparing 2 files string by string Post 302546253 by agama on Wednesday 10th of August 2011 09:07:09 PM
Old 08-10-2011
Glad it's working! To answer your questions first:
Quote:
can I directly insert my fields as ......
awk -F "," -v c1=${31:-31} -v c2=${36:-36} '....and run the script as
./test_sh.........which will read the data from the file script_test.txt......????
Yes, but you will need to redirect your file in like this:
Code:
./test_sh <script_test.txt

Quote:
1) I have to cat my file and put filter for $49....as this field had 4 values.....1,2,3,4..........and then I have to process the above entire code for each value respectively.....
so can I use this code like......

cat scripts_test.txt | awk -F "," '{if ($49==1) print $0}' | CODE........????
Yes, but you don't need cat. Awk can read the file directly so something like this:

Code:
awk -F "," '{if ($49==1) print $0}' scripts_test.txt| CODE........????

However, you can have the programme that generates the matrix make a single pass across your input data and generate all 4 (or however many different values $49 has) matrices. Given that, the script below will print one matrix for each unique value in $49. Thus, there is no need to run the preprocess script that prints only records where $49 == 1 etc.

It will take from the command line, in this order, the column to print vertically, the column to print horizontally, the column which identifies the matrix. If no parms are given, then the defaults are 31 (vert) 36 (horiz) and 49 (id column).

Finally, in the special case where c1 has the value of 255, and $13 is non-zero, the count will be updated as though the value of c1 was zero.

Code:
#!/usr/bin/env ksh

# $1 == c1 == vertical
# $2 == c2 == horizontal
# $3 == matrix id col (midc)
#
awk -F "," -v ct=$ct -v c1=${1:-31} -v c2=${2:-36} -v midc=${3:-49} '
    function sort_order(  what, l,  i, j )
    {
        for( i = 0; i < l; i++ )
        {
            big = 0;
            for( j = 1; j < l-i; j++ )
                if( what[j] > what[big] )
                    big = j;
            if( big != j-1 )
            {
                hold = what[j-1];
                what[j-1] = what[big];
                what[big] = hold;
            }
        }
    }

    {
        mid = $(midc);                      # pick up the matrix id
        if( ! mseen[mid]++ )
            order_m[moidx++] = mid;         # capture for print at end

        if( $(c2) == "" )                   # if value in c2 is missing, we dont count
            next;

        if( $(c1) == "" )
            $(c1) = "BLANK";                # easy eycatcher for empty field

        if( !seen_1[$(c1)]++ )
            order_1[o1idx++] = $(c1);       # order each c1 was observed

        if( !seen_2[$(c2)]++ )
            order_2[o2idx++] = $(c2);       # order each c2 was observed

        if( $(c1) == 255 && $13 != 0  )     # special case: c1 is 255 and $13 is non-zero, count as though c1 was 0
            $(c1) = 0;

        count[mid,$(c1),$(c2)]++;           # count the number of times the pair (c1,c2) appear together
    }

    function print_matrix( mid,     j, i )
    {
        printf( "%15s ", "COLA/COLB" );         # print the header line using the order_2 list
        for( j=0; j < o2idx; j++ )
            printf( "%15s ", order_2[j] );      # %15s will align columns based on a width of 15
        printf( "\n" );                         # new line for first row of matrix

        for( i = 0; i < o1idx; i++ )            # print matrix -- for each row (c1 value)
        {
            printf( "%15s ", order_1[i] );      # print the c1 value (again width of 15)
            for( j=0; j < o2idx; j++ )          # for each column (c2 values)
                printf( "%15d ", count[mid,order_1[i],order_2[j]] );    # print each (width 15 again)

            printf( "\n" );                     # end the row by printing a newline
        }
    }

    END {
        sort_order( order_1, length( order_1 ) );       # sort values from c1, and c2
        sort_order( order_2, length( order_2 ) );
        sort_order( order_m, length( order_m ) );

        for( i = 0; i < moidx; i++ )
        {
            printf( "\nMatrix: %s\n", order_m[i] );     # header for the matrix
            print_matrix( order_m[i] );
        }
    }
'

exit

Assuming you save this file in /home/userx/bin/gen_matrix.ksh and your test data is always in /usr2/data/matrix_input.csv, then you can schedule your job in cron with this command:

Code:
/home/userx/bin/gen_matrix.ksh /usr2/data/matrix_input.csv >/tmp/matrix.out

It will always write the output to the same file. If your input file will be different each day (maybe having the date), or you want your output file to be diffent each day (again with the date), then you will need to construct your input and output filenames in the script and use them on the awk command. Your cron command then just consists of the name of the script.

Examples of this:
Code:
#!/usr/bin/env ksh

date=$( date "+%Y%m%d" )   # date in yyyymmdd (mmdd sorts in order where ddmm does not)
input_file=/usr2/data/matrix_input_$date.csv   # build your file names 
output_file=/usr2/output/matrix_out.$date

# run the awk programme using your filenames on the last line
awk -F "," -v ct=$ct -v c1=${1:-31} -v c2=${2:-36} -v midc=${3:-49} '


#### body of awk programme ######


' $input_file >$output_file    # read from, write to, filenames you created

Hope this gets you further along.

Last edited by agama; 08-10-2011 at 10:08 PM.. Reason: typo
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed problem - replacement string should be same length as matching string.

Hi guys, I hope you can help me with my problem. I have a text file that contains lines like this: 78 ANGELO -809.05 79 ANGELO2 -5,000.06 I need to find all occurences of amounts that are negative and replace them with x's 78 ANGELO xxxxxxx 79... (4 Replies)
Discussion started by: amangeles
4 Replies

2. Shell Programming and Scripting

Extracting a string from one file and searching the same string in other files

Hi, Need to extract a string from one file and search the same in other files. Ex: I have file1 of hundred lines with no delimiters not even space. I have 3 more files. I should get 1 to 10 characters say substring from each line of file1 and search that string in rest of the files and get... (1 Reply)
Discussion started by: mohancrr
1 Replies

3. Shell Programming and Scripting

problem in comparing numeric with string

Hi all, I am having a problem in comparing numeric value with string. I have a variable in my script which gets the value dynamically. It can be a numeric value or a string. I have to do separate task based on its value numeric or sting variable VARIABLE. I grep FILE_COUNT and obtained... (7 Replies)
Discussion started by: naren_0101bits
7 Replies

4. Shell Programming and Scripting

Problem comparing String using IF stmt

Hi frnds Im facing an issues while trying to compare string using IF stmt, my code is: chkMsgName=`Service Fee Detail` if then if then if then echo "Valid File Ready for processing" fi fi ... (5 Replies)
Discussion started by: balesh
5 Replies

5. Shell Programming and Scripting

Parsing a long string string problem for procmail

Hi everyone, I am working on fetchmail + procmail to filter mails and I am having problem with parsing a long line in the body of the email. Could anyone help me construct a reg exp for this string below. It needs to match exactly as this string. GetRyt... (4 Replies)
Discussion started by: cwiggler
4 Replies

6. UNIX for Dummies Questions & Answers

Comparing a String variable with a string literal in a Debian shell script

Hi All, I am trying to to compare a string variable with a string literal inside a loop but keep getting the ./testifstructure.sh: line 6: #!/bin/sh BOOK_LIST="BOOK1 BOOK2" for BOOK in ${BOOK_LIST} do if then echo '1' else echo '2' fi done Please use next... (1 Reply)
Discussion started by: daveu7
1 Replies

7. Shell Programming and Scripting

grep exact string from files and write to filename when string present in file

I am attempting to grep an exact string from a series of files within a directory and append that output to the filename when it is present in the file. I've been after this all day with no luck. Thanks for your help in advance :wall:. (4 Replies)
Discussion started by: JC_1
4 Replies

8. Shell Programming and Scripting

How to append a string by comparing another string?

Hi , I have one file like BUD,BDL BUDCAR BUD,BDL BUDLAMP ABC,CDF,KLT ABISKAR ABC,CDF,KLT CORNEL ABC,CDF,KLT KANNAD JKL,HNM,KTY,KJY JAGAN JKL,HNM,KTY,KJY HOUSE JKL,HNM,KTY,KJY KATAK JKL,HNM,KTY,KJY KOLKA The o/p should be like BUD,BDL BUDCAR,BUDLAMP ABC,CDF,KLT... (4 Replies)
Discussion started by: jagdishrout
4 Replies

9. Shell Programming and Scripting

Grep string in files and list file names that contain the string

Hi, I have a list of zipped files. I want to grep for a string in all files and get a list of file names that contain the string. But without unzipping them before that, more like using something like gzcat. My OS is: SunOS test 5.10 Generic_142900-13 sun4u sparc SUNW,SPARC-Enterprise (8 Replies)
Discussion started by: apenkov
8 Replies

10. UNIX for Advanced & Expert Users

Help comparing string, please

Good morning, I need compare this string. if || || ; then But this line not work, somebody can say me what is the error. Thank you for advanced. (5 Replies)
Discussion started by: systemoper
5 Replies
All times are GMT -4. The time now is 04:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy