Shell Programming and Scripting

Hi All,

This is very urgent, I've a data file with 1.7 millions rows in the file and the delimiter is cedilla and I need to format the data in such a way that if the NF in the next row is less than 1, it will append that value to previous line.

Any help will be appricated.

Thanks,
cumeh1624

---------- Post updated at 10:50 PM ---------- Previous update was at 10:46 PM ----------

I will prefer any suggestion using awk command because of performance

By NF < 1, do you mean the row is empty? (a blank line, perhaps)

Code:

 awk '{printf "%s",(NF>0?$0:"\n")}' filename

It means the very line has one field without cedilla as a field seperator or it has a blank line.

Code:

awk 'BEGIN{FS='Ç'}
  NR == 1 {p = $0; next}
  NF > 1 {print p; p = $0}
  NF <= 1 {p = (p " " $0)}
  END {print p}' input.txt > output.txt

Hi Srinishoo,

The script works fine but there is a limitation to it. I tested it with 1.7 million rows in a data file and about 10,471 rows with numerical values and some few characters with 5 to 9 fields didn't get loaded into the output file.

So when processing the files this statement "

Code:

NF > 1 {print p; p = $0}

" didn't send about 10,471 rows to the output file and the only different with other rows is that it contain almost numerica values in most of the column fields.

This is examplle of what one of the row that didn't get loaded looks like

Code:

29863Ç890000000Ç543209911ÇCHNGOHGÇ000000001Ç055Ç

do you hav a suggestion in handling this issue?

That should not be the case. Try the below code
exchanged single quotes with double quotes for FS assignment.

Code:

awk 'BEGIN{FS = "Ç"}
  NR == 1 {p = $0; next}
  NF > 1 {print p; p = $0}
  NF <= 1 {p = (p " " $0)}
  END {print p}' input.txt > output.txt

The number of rows will change between input and output as you are append few lines to previous line
and verify if both the input has the row you have provided

Code:

awk '$0 ~ /29863Ç890000000Ç543209911ÇCHNGOHGÇ000000001Ç055Ç/' input.txt
awk '$0 ~ /29863Ç890000000Ç543209911ÇCHNGOHGÇ000000001Ç055Ç/' output.txt