Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Reconciling two CSV files using shell scripting Post 303041043 by Neo on Tuesday 12th of November 2019 08:34:47 PM
Old 11-12-2019
You can easily do this with PHP, for example, you can read the CSV files into arrays and and compare the arrays:


Quick examples to get you started (untested, for example only):

Code:
<?php
$csvFile1 = file('../somefile.csv');
$csvarray1 = [];
 foreach ($csvFile1 as $line) {
     $csvarray1[] = str_getcsv($line);
 }

$csvFile2 = file('../someotherfile.csv');
$csvarray2 = [];
 foreach ($csvFile2 as $line) {
     $csvarray2[] = str_getcsv($line);
 }

Then you can use one or two of myriad PHP array methods to check for existence of keys, differences in arrays, etc. See, for example, these methods:

Code:
<?php
array_diff();
array_keys();
array_diff_ukey();
array_key_exists()

Then, after you have your new array as you desire, then you can simply convert your temporary array back to PHP, for example:

Code:
<?php
fputcsv ();

In a nutshell, is easy to process CSV files with PHP either from a script, directly from the command line, or interactively from the command line; most notably converting CSV files to arrays, doing array operations, and converting back to a CSV file.

So personally I would do this in PHP and not use shell scripts because PHP is built to do this kind of processing easily.

OBTW, these days I tent to quickly prototype and test my PHP ideas interactively in the shell as follows:


Code:
php -a

Then in the shell in interactive mode, I test and debug logic quickly and easily.

This is how I process CSV files. You can also easily to this same type of CSV processing easily in Python, BTW.

Other may have more "shell script-like" approaches for you which do not use PHP or Python; I am only describing how I approach these types of issues in CSV, JSON or other standard file formats. Since most of my work all touches the Internet somehow (web servers), and those servers are mostly PHP based, I like to stick to code I can reuse and debug together, so that is why I tend to use PHP over Python. Actually, if my apps were not mostly PHP based, I would use Python more.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with shell scripting for accepting .csv files as CLA

I want to automate test script on shell scripting. There are 2 .csv files named account.csv and balance.csv.These files needs to passed as command line arguments and the following logic needs to applied further. Any account with a balance that was due before Oct 23, 2007 has an overdue... (2 Replies)
Discussion started by: coolguy123
2 Replies

2. Shell Programming and Scripting

How to insert a sequence number column inside a pipe delimited csv file using shell scripting?

Hi All, I need a shell script which could insert a sequence number column inside a dat file(pipe delimited). I have the dat file similar to the one as shown below.. |A|B|C||D|E |F|G|H||I|J |K|L|M||N|O |P|Q|R||S|T As shown above, the column 4 is currently blank and i need to insert sequence... (5 Replies)
Discussion started by: nithins007
5 Replies

3. Shell Programming and Scripting

Shell scripting:from text file to CSV

Hello friends, I have a file as follows: "empty line" content1 content2 content3 content1 content2 content3 content1 content2 content3 It starts with an empty line, how can i get a csv like this: (12 Replies)
Discussion started by: kraterions
12 Replies

4. Shell Programming and Scripting

How to create or convert to pdf files from csv files using shell script?

Hi, Can anyone help me how to convert a .csv file to a .pdf file using shell script Thanks (2 Replies)
Discussion started by: ssk250
2 Replies

5. Shell Programming and Scripting

How to calculate avg values of csv file using shell scripting .?

hi all i have a reporting work and i want it to be automated using shell scripting kindly let me know how can i make that possibe . eg data are :... (2 Replies)
Discussion started by: Avinash shaw
2 Replies

6. Shell Programming and Scripting

How to calculate average of csv using shell scripting?

Hi, I need to calculate the average of the following values using shell scripitng. Can anyone please suggest a solution? ... (10 Replies)
Discussion started by: karan pratap si
10 Replies

7. Shell Programming and Scripting

Shell script for field wise record count for different Files .csv files

Hi, Very good wishes to all! Please help to provide the shell script for generating the record counts in filed wise from the .csv file My question: Source file: Field1 Field2 Field3 abc 12f sLm 1234 hjd 12d Hyd 34 Chn My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies

8. Shell Programming and Scripting

Need a piece of shell scripting to remove column from a csv file

Hi, I need to remove first column from a csv file and i can do this by using below command. cut -f1 -d, --complement Mytest.csv I need to implement this in shell scripting, Whenever i am using the above command alone in command line it is working fine. I have 5 files in my directory and... (3 Replies)
Discussion started by: Samah
3 Replies

9. Shell Programming and Scripting

Read csv file, convert the data and make one text file in UNIX shell scripting

I have input data looks like this which is a part of a csv file 7,1265,76548,"0102:04" 8,1266,76545,"0112:04" I need to make the output data should look like this and the output data will be part of text file: 7|1265000 |7654899 |A| 8|12660000 |76545999 |B| The logic behind the... (6 Replies)
Discussion started by: RJG
6 Replies

10. UNIX for Beginners Questions & Answers

Export Oracle multiple tables to multiple csv files using UNIX shell scripting

Hello All, just wanted to export multiple tables from oracle sql using unix shell script to csv file and the below code is exporting only the first table. Can you please suggest why? or any better idea? export FILE="/abc/autom/file/geo_JOB.csv" Export= `sqlplus -s dev01/password@dEV3... (16 Replies)
Discussion started by: Hope
16 Replies
ARRAY_INTERSECT_UASSOC(3)						 1						 ARRAY_INTERSECT_UASSOC(3)

array_intersect_uassoc - Computes the intersection of arrays with additional index check, compares indexes by a callback function

SYNOPSIS
array array_intersect_uassoc (array $array1, array $array2, [array $...], callable $key_compare_func) DESCRIPTION
array_intersect_uassoc(3) returns an array containing all the values of $array1 that are present in all the arguments. Note that the keys are used in the comparison unlike in array_intersect(3). PARAMETERS
o $array1 - Initial array for comparison of the arrays. o $array2 - First array to compare keys against. o $... - Variable list of array arguments to compare values against. o $key_compare_func - The comparison function must return an integer less than, equal to, or greater than zero if the first argument is considered to be respectively less than, equal to, or greater than the second. int callback (mixed $a, mixed $b) RETURN VALUES
Returns the values of $array1 whose values exist in all of the arguments. EXAMPLES
Example #1 array_intersect_uassoc(3) example <?php $array1 = array("a" => "green", "b" => "brown", "c" => "blue", "red"); $array2 = array("a" => "GREEN", "B" => "brown", "yellow", "red"); print_r(array_intersect_uassoc($array1, $array2, "strcasecmp")); ?> The above example will output: Array ( [b] => brown ) SEE ALSO
array_intersect(3), array_intersect_assoc(3), array_uintersect_assoc(3), array_uintersect_uassoc(3), array_intersect_key(3), array_inter- sect_ukey(3). PHP Documentation Group ARRAY_INTERSECT_UASSOC(3)
All times are GMT -4. The time now is 04:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy