Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Reconciling two CSV files using shell scripting Post 303041052 by RudiC on Wednesday 13th of November 2019 04:14:28 AM
Old 11-13-2019
How about
Code:
awk -F, '
FNR == 1        {FCNT++
                }
FNR > 1         {KEYS[$1]
                 for (i=2; i<=NF; i++) W[$1,FCNT,i] = $i
                }
END             {printf "Key, diff columns"
                 for (i=1; i<NF; i++) printf ",f1_value%d,f2_value%d,value%d_diff", i, i, i
                 printf RS
                 for (k in KEYS)        {for (i=NF; i>1; i--)   {N1 = W[k,1,i]
                                                                 N2 = W[k,2,i]
                                                                 D1 = N2 - N1
                                                                 OUT = sprintf (",%s,%s,%s", N1, N2, D1) OUT
                                                                 if (D1) COLS = ",Value" i-1 COLS
                                                                }
                                         print k, "\"" substr (COLS,2) "\"", substr (OUT, 2)
                                         OUT = COLS = ""
                                        }
                }
' OFS=, SUBSEP=, file[12]
Key, diff columns,f1_value1,f2_value1,value1_diff,f1_value2,f2_value2,value2_diff,f1_value3,f2_value3,value3_diff,f1_value4,f2_value4,value4_diff
A,"Value2,Value3,Value4",50,50,0,100,80,-20,50,45,-5,40,50.....,10


Last edited by RudiC; 11-13-2019 at 05:20 AM..
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with shell scripting for accepting .csv files as CLA

I want to automate test script on shell scripting. There are 2 .csv files named account.csv and balance.csv.These files needs to passed as command line arguments and the following logic needs to applied further. Any account with a balance that was due before Oct 23, 2007 has an overdue... (2 Replies)
Discussion started by: coolguy123
2 Replies

2. Shell Programming and Scripting

How to insert a sequence number column inside a pipe delimited csv file using shell scripting?

Hi All, I need a shell script which could insert a sequence number column inside a dat file(pipe delimited). I have the dat file similar to the one as shown below.. |A|B|C||D|E |F|G|H||I|J |K|L|M||N|O |P|Q|R||S|T As shown above, the column 4 is currently blank and i need to insert sequence... (5 Replies)
Discussion started by: nithins007
5 Replies

3. Shell Programming and Scripting

Shell scripting:from text file to CSV

Hello friends, I have a file as follows: "empty line" content1 content2 content3 content1 content2 content3 content1 content2 content3 It starts with an empty line, how can i get a csv like this: (12 Replies)
Discussion started by: kraterions
12 Replies

4. Shell Programming and Scripting

How to create or convert to pdf files from csv files using shell script?

Hi, Can anyone help me how to convert a .csv file to a .pdf file using shell script Thanks (2 Replies)
Discussion started by: ssk250
2 Replies

5. Shell Programming and Scripting

How to calculate avg values of csv file using shell scripting .?

hi all i have a reporting work and i want it to be automated using shell scripting kindly let me know how can i make that possibe . eg data are :... (2 Replies)
Discussion started by: Avinash shaw
2 Replies

6. Shell Programming and Scripting

How to calculate average of csv using shell scripting?

Hi, I need to calculate the average of the following values using shell scripitng. Can anyone please suggest a solution? ... (10 Replies)
Discussion started by: karan pratap si
10 Replies

7. Shell Programming and Scripting

Shell script for field wise record count for different Files .csv files

Hi, Very good wishes to all! Please help to provide the shell script for generating the record counts in filed wise from the .csv file My question: Source file: Field1 Field2 Field3 abc 12f sLm 1234 hjd 12d Hyd 34 Chn My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies

8. Shell Programming and Scripting

Need a piece of shell scripting to remove column from a csv file

Hi, I need to remove first column from a csv file and i can do this by using below command. cut -f1 -d, --complement Mytest.csv I need to implement this in shell scripting, Whenever i am using the above command alone in command line it is working fine. I have 5 files in my directory and... (3 Replies)
Discussion started by: Samah
3 Replies

9. Shell Programming and Scripting

Read csv file, convert the data and make one text file in UNIX shell scripting

I have input data looks like this which is a part of a csv file 7,1265,76548,"0102:04" 8,1266,76545,"0112:04" I need to make the output data should look like this and the output data will be part of text file: 7|1265000 |7654899 |A| 8|12660000 |76545999 |B| The logic behind the... (6 Replies)
Discussion started by: RJG
6 Replies

10. UNIX for Beginners Questions & Answers

Export Oracle multiple tables to multiple csv files using UNIX shell scripting

Hello All, just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table. Can you please suggest why? or any better idea? export FILE="/abc/autom/file/geo_JOB.csv" Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
DBD::File(3)						User Contributed Perl Documentation					      DBD::File(3)

NAME
DBD::File - Base class for writing DBI drivers SYNOPSIS
This module is a base class for writing other DBDs. It is not intended to function as a DBD itself. If you want to access flatfiles, use DBD::AnyData, or DBD::CSV, (both of which are subclasses of DBD::File). DESCRIPTION
The DBD::File module is not a true DBI driver, but an abstract base class for deriving concrete DBI drivers from it. The implication is, that these drivers work with plain files, for example CSV files or INI files. The module is based on the SQL::Statement module, a simple SQL engine. See DBI for details on DBI, SQL::Statement for details on SQL::Statement and DBD::CSV or DBD::IniFile for example drivers. Metadata The following attributes are handled by DBI itself and not by DBD::File, thus they all work like expected: Active ActiveKids CachedKids CompatMode (Not used) InactiveDestroy Kids PrintError RaiseError Warn (Not used) The following DBI attributes are handled by DBD::File: AutoCommit Always on ChopBlanks Works NUM_OF_FIELDS Valid after "$sth-"execute> NUM_OF_PARAMS Valid after "$sth-"prepare> NAME Valid after "$sth-"execute>; undef for Non-Select statements. NULLABLE Not really working, always returns an array ref of one's, as DBD::CSV doesn't verify input data. Valid after "$sth-"execute>; undef for Non-Select statements. These attributes and methods are not supported: bind_param_inout CursorName LongReadLen LongTruncOk Additional to the DBI attributes, you can use the following dbh attribute: f_dir This attribute is used for setting the directory where CSV files are opened. Usually you set it in the dbh, it defaults to the current directory ("."). However, it is overwritable in the statement handles. f_ext This attribute is used for setting the file extension where (CSV) files are opened. There are several possibilities. DBI:CSV:f_dir=data;f_ext=.csv In this case, DBD::File will open only "table.csv" if both "table.csv" and "table" exist in the datadir. The table will still be named "table". If your datadir has files with extensions, and you do not pass this attribute, your table is named "table.csv", which is probably not what you wanted. The extension is always case-insensitive. The table names are not. DBI:CSV:f_dir=data;f_ext=.csv/r In this case the extension is required, and all filenames that do not match are ignored. f_schema This will set the schema name. Default is the owner of the folder in which the table file resides. "undef" is allowed. my $dbh = DBI->connect ("dbi:CSV:", "", "", { f_schema => undef, f_dir => "data", f_ext => ".csv/r", }) or die $DBI::errstr; The effect is that when you get table names from DBI, you can force all tables into the same (or no) schema: my @tables $dbh->tables (); # no f_schema "merijn".foo "merijn".bar # f_schema => "dbi" "dbi".foo "dbi".bar # f_schema => undef foo bar Driver private methods data_sources The "data_sources" method returns a list of subdirectories of the current directory in the form "DBI:CSV:f_dir=$dirname". If you want to read the subdirectories of another directory, use my ($drh) = DBI->install_driver ("CSV"); my (@list) = $drh->data_sources (f_dir => "/usr/local/csv_data" ); list_tables This method returns a list of file names inside $dbh->{f_dir}. Example: my ($dbh) = DBI->connect ("DBI:CSV:f_dir=/usr/local/csv_data"); my (@list) = $dbh->func ("list_tables"); Note that the list includes all files contained in the directory, even those that have non-valid table names, from the view of SQL. KNOWN BUGS
o The module is using flock () internally. However, this function is not available on all platforms. Using flock () is disabled on MacOS and Windows 95: There's no locking at all (perhaps not so important on MacOS and Windows 95, as there's a single user anyways). AUTHOR
This module is currently maintained by H.Merijn Brand < h.m.brand at xs4all.nl > and Jens Rehsack < rehsack at googlemail.com > The original author is Jochen Wiedmann. COPYRIGHT AND LICENSE
Copyright (C) 2009 by H.Merijn Brand & Jens Rehsack Copyright (C) 2004 by Jeff Zucker Copyright (C) 1998 by Jochen Wiedmann All rights reserved. You may freely distribute and/or modify this module under the terms of either the GNU General Public License (GPL) or the Artistic License, as specified in the Perl README file. SEE ALSO
DBI, Text::CSV, Text::CSV_XS, SQL::Statement perl v5.12.1 2009-06-05 DBD::File(3)
All times are GMT -4. The time now is 06:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy