Shell Programming and Scripting

+
Hi bakunin,

I used the command

Code:

cat -e file.txt

to find out the EOL character.

For issue related lines we have data in Notepad++(see the attachment Delimeter_Issue_notepad++1) having "LF" as the EOL and Unix we have"$" (see the attachment Delimeter_Issue_Unix1).

After replacing the "LF" with "CR" in Notepad++ we are seeing "CR" (see the attachment Delimeter_Issue_notepad++2) and in Unix we have "^M"(see the attachment Delimeter_Issue_Unix1).

I think if we replace the

Code:

"$"

with

Code:

"^M"

issue will resolve.
Could you please let me know is there any way to do this.
Or could you please let me know the best approach to fix this issue.

Thanks,
Narasimha

DOS line terminators (<CR>, \r, ^M, 0x0D) in *nix system are definitely in the wrong spot. Don't use them, and, less than ever, ADD them! DON'T use notepad to create files to be used/analysed on *nix systems.

The LF char is used in EXCEL to mark a line break within a cell. Does that file come from EXCEL?

Hi RudiC,

No this file is not coming from EXCEL.We have donloaded the file from website and in manual read they suggested one command to fix this kind of issues.
We tried that command but no use.

Below is the description from the Manual file which they have provided.

Several files contain records that span multiple lines. This often causes problems when importing into relational databases. Users may wish to remove such features from a file before attempting to import its contents. For example, the following awk command can be used (on Linux or MacOS platforms) to address some of these situations.

Code:

awk 'BEGIN {FS="|";} {if ((length($2)==11) && index($2,"NCT") !=0) printf "\n%s",$0; else printf "%s",$0;}' arm_groups.txt | sed -e “s/[[:space:]]\+/ /g” > arm_groups.out

This command looks at each line in the arm_groups.txt file and determines if the 2nd field is the NCT_ID (length is 11 and first 3 chars are ‘NCT') which suggests it represents an actual record (as opposed to ‘carry-over text'). If so, it prints the record on a new line. Subsequent lines that do not have an NCT_ID in the second field are assumed to be carry-over text and are appended to this record. The ‘sed' clause near the end of the command

Code:

(sed -e "s/[[:space:]]\+/ /g")

simply compresses contiguous spaces into a single space.

Thanks,
Narasimha

There's no NCT_ID in either of your samples. Why do you send us mess around with incorrect sample data and irrelevant approaches when there's a proven solution that might fail in your special case?

actually: no. The "$" is just signifying the line end.

The problem you are obviously encountering is the old DOS<->UNIX problem:

in DOS lines are separated by the <CR><LF>-character sequence. That is, if you see a file (in DOS/Windows) like:

Code:

AB
CD

This file has in fact 6 bytes: "<A><B><CR><LF><C><D>". CR (Carriage Return) and LF (Line Feed) were originally printer-steering characters and this way DOS did circumvent the necessity to implement a printing program which (in professional OSes) entered these control sequences. Instead in DOS "printing" meant just dumping the file at it was to the printer device.

In Unix the situation was different and indeed it had such a printing system. Therefore it was not necessary to have a two-character sequence to separate lines and hence UNIX systems have only a single character "NL" (new line) to separate lines. Incidentally it is the same character as the "LF" in DOS, which is why you see the additional "^M" character at the end of the line. These are simply the second of the CR-LF pair. The file above in UNIX would consist also of 6 characters, but only because proper UNIX files have <EOF> (End Of File) character at their end:"<A><B><NL><C><D><EOF>".

Your problem comes most probably from transferring files back and forth between DOS- (or Windows-) systems and UNIX-systems without properly translating between them. ftp, for instance, has two modes: A(scii) and BI(nary): binary means no such translation takes place. ASCII means the ftp client becomes aware on which system it runs and to which system it transfers files and translates these line endings to what is proper on the target system. Alas, some email clients base their automatic detection on file-names (like "*.txt", etc.) and many users (you, obviously, included) don't know how and/or when to set the correct mode. This is why these ill-formed files happen.

You can either remove the superfluous line-ending characters in UNIX via the givem sed-script (you have to do that PRIOR to all the other scripts) or you can use the dos2unix and unix2dos utilities (which do the same, just in a "prepackaged" way) or you can use (on some systems) the recode-command, which also does the same.

I hope this helps.

bakunin

Isn't the <EOF> char actually just another <LF>?

The EOF character is from CP/M. (Where the directory only contained blocks but not the byte-length, so the end in the last block was marked with an EOF.)
Neither MS-DOS/FAT nor Unix need the EOF character. Early MS-DOS applications used it for compatibility with CP/M.
The LF (or NL) character should be appended to the last line of a Unix text file, like is appended to the previous lines. This is a convention. If missing, some text processing utilities could skip the last line or give a warning.