Shell Programming and Scripting

Hi all,

We get data in the form of file(1.5gb) (*.csv) and it is fixed length file with 5 delimiters & 5 columns
for example

Code:

rec1_1<;|rec1_2<;|rec1_3<;|rec1_4<;|rec1_5<;|

the above rec1 is perfect example that our process accepts it for execution.

but for few records we are now getting some
1) part of data in new line
2) that part starts with Tabs(x3 tabs or x6 tabs)

These Tabs are unwanted and the split should not take place.

the example records are:

Code:

 rec2_1<;|rec2_2<;|rec2_3<;|rec2_4<;|
    <;|
    rec3_5
<;|
    rec4_5 <;|
ISX_SEARCH_FILTER_WINDOW<;|E<;|A<;|0000<;|
       Type      : Combinatie van foutc<;|

Here in this the last record is real example but somehow Tabs are replaced with spaces here.

we now want to exclude such records before processing them and capture them in some file.

can any one help me
1) why those splits are taking in new line
2) why unnecessary tabs are embedded with the other parth of recor in new line
3) & How to identify those records and direct them to a reject file.

We Access those records with NLS_lang = UTF-16LE without which it is immpossible to see the data.

Thanks

Presumably, the file you are processing came from somewhere. You need to examine the source of that data to determine why it is incorrectly formatted. You have given us absolutely no information that would enable us to determine why the source of your data is incorrectly formatting your data.
Presumably, the file you are processing came from somewhere. You need to examine the source of that data to determine why it is incorrectly formatted. You have given us absolutely no information that would enable us to determine why the source of your data is incorrectly formatting your data.

Code:

awk -F '<;[|]' 'NF!=6' input_file.csv > malformed_records_file.csv

What operating system are you using? On UNIX and Linux and BSD based systems, you might want something like LC_CTYPE=UTF-16LE. Setting an environment variable with spaces around the equals-sign is not correct syntax for shell commands. Maybe there is something else going on here that would help us understand what environment is being used to process your data???

Moderator's Comments:

Putting this to "Shell Scripting", where it belongs.

bakunin