Shell Programming and Scripting

Hi All,

I want to find the difference between two files, by checking only the headers (column names) and report if any new column is added in the latest file.

For Ex: If the file "declartion.txt has these columns

Code:

url;image;id;showcase_id;showcase_name

and the actual file "feed.txt" has a newly added column either in the middle or in the last like

Code:

url;image;id;showcase_id;colour;showcase_name

The declaration file which will have only the header (column names), but the "feed.txt" will have column names and the values for those columns under each column. Comparison is required only for the column names (header) and not any values.

After the comparison the shell script should should report the newly added column For Ex: "colour" in a file called "report_column_mis-match.txt". Once the report file is created I the shell script should further remove the new column which is added by the data provider along with it's values.

The data provider is adding columns each now and then not informing us in advance which is creating the problem while loading the file in to mysql database. since the values get loaded haywire when the new column added in the data file but the same is not added in the mysql table.

I finding this bit complicated since I am not a shell script expert. Please help.

Thanks in advance.

Regards
Praveen

Moderator's Comments:

Please use CODE tags as in the Forum Rules for code, files, input & output/errors They make it clearer to read and preserves multiple spaces and other critical information about the content

What have you tried so far?

Hi Don Cragun,

I am still doing some research on this to get the clue.

Regards
Praveen

---------- Post updated at 04:46 AM ---------- Previous update was at 03:33 AM ----------

I am able to read the first line from feed.txt and compare it with declaration.txt and print the difference.

But I am still not able to find the position of the column and its values and remove it. Not sure how to handle multiple new columns and remove it, if found.

Please help.

Thanks & regards
Praveen

Hi Praveen,

Just a couple of quick questions, does the column always appear in the same place along with it's data.

Can you show us an example of the data?

Regards

Dave

Take this as a starting point:

Code:

awk -F\; 'NR==FNR               {split ($0, HD)
                                 DEC=$0
                                 next
                                }
          FNR==1 && $0==DEC     {exit}
          FNR==1                {for (i=1; i<=NF && $i==HD[i]; i++);
                                 print "Column " i ": " $i > "mismatch.txt"}
                                {$i=""; sub (/;;/,";"); sub (/^;|;$/, "")
                                }
          1
        ' OFS=";" declartion.txt feed.txt

Hi Rudic,

But it is not listing the newly added column. If the above script is run, it is giving the output as

Column 1: url,dealerid,dealer_name,location,zip,make

in mismatch.txt

As explained above, my need is to find the new columns (in this case two new columns url and make) and remove them from the feed.txt along with it's respective values.

Thanks & Regards
Praveen

Well, it does for the files you gave in your spec in post #1. With declartion.txt

Code:

url;image;id;showcase_id;showcase_name

and feed.txt

Code:

url;image;id;showcase_id;colour;showcase_name
url1;image1;id1;showcase_id1;colour1;showcase_name1
url2;image2;id2;showcase_id2;colour2;showcase_name2
url3;image3;id3;showcase_id3;colour3;showcase_name3
url4;image4;id4;showcase_id4;colour4;showcase_name4

it has stdout as

Code:

url;image;id;showcase_id;showcase_name
url1;image1;id1;showcase_id1;showcase_name1
url2;image2;id2;showcase_id2;showcase_name2
url3;image3;id3;showcase_id3;showcase_name3
url4;image4;id4;showcase_id4;showcase_name4

and mismatch.txt as

Code:

Column 5: colour

As explained above, it does what was specified, and you obviously changed the input file structure.