Shell Programming and Scripting

Hi All,

I've two .csv files as below
file1.csv
abc, tdf, 223, tpx
jgsd, tex, 342, rpy
a, jdjdsd, 423, djfkld

Where as file2.csv is the new version of file1.csv with some added fields in the end of each line and some additional lines.
lfj, eru, 98, jkldj, 39, jdkj9
abc, tdf, 223, tpx, 4sdd, 43
jgsd, tex, 342, rpy, 343js, j88
a, jdjdsd, 423, djfkld, djd, 322i
djlfj, djd, 499, djld, 323u, d88l
fjdkl, jdfsld, 45k, djdkl, 343, 334p

Now my requirement is to get only these additional lines from file2.csv.
I tried with 'diff' and 'comm'. Due to added additional fields I am not able to get the desired output as below:
lfj, eru, 98, jkldj, 39, jdkj9
djlfj, djd, 499, djld, 323u, d88l
fjdkl, jdfsld, 45k, djdkl, 343, 334p

Even I am still trying to solve this myself. Any help would be much appreciated from any of friends...

Thanks in advance / Mysore Ganapati

You can try something like this, but it may fail in some situations.

Code:

fgrep -f file1.csv -v file2.csv

I gave you the sample of files. But in reality, both files are having 10 millions of records. When I tried your solution using fgrep, got the below error message:
fgrep: could not allocate memory for wordlist

Are there any other solutions please...

Hm,
if the number of columns in the first csv is fixed,
you could try Awk:

Code:

awk -F, 'NR == FNR {
  _[$0]
  next
  }
!(($1 FS $2 FS $3 FS $4) in _)
' file1.csv file2.csv

But you may encounter similar problems ...

Or, something like this, will not be fast :

Code:

awk -F, 'BEGIN {
  f = ARGV[1]
  ARGV[1] = ""
  }
{ 
  while ((getline _f < f) > 0)
    if (($1 FS $2 FS $3 FS $4) == _f) 
	  next
	close(f)
  }
1' file1.csv file2.csv

You could modify the second example and use it even if the number of columns in the first file is variable.