Shell Programming and Scripting

Hi All,

There are list of 4-5 .csv files which has 12 columns.In some cases one of the record is split into 2 records. What needs to be done is this split record has to be auto corrected and placed in the csv file.

Eg:

Let us consider sample.csv file and in normal conditions the file would like below.

Code:

a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n

In some cases the sample.csv file would look like

Code:

a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,
i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n

Code:

a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h
,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,n

If you see the line 2,3 the record is broken. It should be a single record.


How to identify the broken records?
How can we correct the file to look like the normal sample.csv file written earlier?


Thanks,
Karthik

If line are only broken in two and not more, you could try something like this:

Code:

awk 'NF<12{getline p; $0=$0 p}1' FS=, OFS=, file

This should fix the file even if lines are broken into more than two lines:

Code:

perl -0pe 's/,\n/,/g;s/\n,/,/g' file

True, unless there are empty fields in the first or last position, like

Code:

a,b,c,d,g,h,i,j,k,l,m,n
a,b,c,d,g,h,
i,j,k,l,m,n
a,b,c,d,g,h,i,j,k,l,m,
,b,c,d,g,h,i,j,k,l,m,n

--edit--
This may also work for multiple lines and cases of empty fields, if we know it should always be exactly 12 fields per line:

Code:

awk '{while(NF<12){getline p; $0=$0 p}}1' FS=, OFS=, file

I tried the codes which you guys gave. None of them work, all they do is just write the recrods as it is.

All of that works for me.
Hope you are aware that these don't edit the original file, rather output it into the screen, right?

Post the sample of the actual file and the code you have used for us to help you.

--ahamed

Yes yes i'm aware that these dont edit the orignal file.

All i'm doing is redirecting the output of these commands into another file.

I cant get the actual code. But it is exactly the same as the sample code i mentioned. It has 12 columns, separated by a comma.