I have a csv (tab delimited) file that is created by an application (that I didn't write).

Every so often it throw out a <U+FEFF> (Zero Width no break space) character at the begining of a tabbed field. The charcater is invisible to some editors, but it shows up bolded in less.

The issue is that these CSV files are being imported in a sql db and it's causing issues. I would like to find a way to scrub this out. As I have learned this is difficult, grep and sed won't find the unicode character (which is apparently UTF-16). I have no control over the originating application. I can only control for the output once it is written to a file.

Is there anything I can do to scrub these out? i already have a shell script that changes the line endings, is there something I can do in the bash?

Not sure it will work, but dos2unix may fix it. Check the man page - dos2unix and unix2dos should be on your system.

Do I understand it correctly that you want to frequently get rid of this character from some *.csv files? In my opinion best would be a simple C/C++ application that reads the file byte-by-byte and writes it byte-by-byte with skipping the unwanted ones. To improve performance you could optimize reads/writes.

That is exactly what I am looking for. I was hoping there was something I could do in the bash, without having to dust off really old c skills (I mean really really old)