awk Command to add Carriage Return and Line Feed


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers awk Command to add Carriage Return and Line Feed
# 8  
Old 10-16-2018
Neither awk nor sed have defined behavior according to the standards if the file being processed is not a text file. If an input file contains any characters and the last character in the file is not a <newline> character, that input file is not a text file. (So the code Scrutinizer suggested also depends on GNU extensions to awk that are not portable and not required by the standards.)

Assuming that all lines in your file are separated by DOS <carriage-return><newline> line separators and you just want to add a <carriage-return><newline> terminator to the last line of a file if it doesn't already exist, and assuming that an empty input file should produce an empty output file (not adding a DOS format empty line), the following POSIX-conforming shell script should be portable to most systems:
Code:
{ cat "${1:-file}"; printf '\r\n'; } | awk '
{	if(empty)
		print "\r"
	if($0 != "\r") {
		print
		empty = 0
	} else	empty = 1
}' > "${2:-newfile}"

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.

This script takes two operands. The first operand is the name of the input file you want to process. The second operand is the name of the output file you want to produce. If neither of these operands are specified on the command line, the input file name defaults to file and the output file name defaults to newfile.
# 9  
Old 10-16-2018
Assuming you are using Unix/Linux and are only interested in appending a line terminator to the last line if it doesn't already exist, you could use ed:
Code:
$ echo wq | ed myfile
Newline appended
18
18
 $


Andrew
# 10  
Old 10-16-2018
Quote:
Originally Posted by Don Cragun
Neither awk nor sed have defined behavior according to the standards if the file being processed is not a text file. If an input file contains any characters and the last character in the file is not a <newline> character, that input file is not a text file. (So the code Scrutinizer suggested also depends on GNU extensions to awk that are not portable and not required by the standards.)
[..]
Hi Don, I see where you are going, I may have read the question wrong, I interpreted it as "add a carriage return is it is missing at the end of a line (which ends in linefeed)". So for that, my suggestion does not use extensions - GNU or otherwise - to the standard. I did not mean to also correct a missing newline at the end of the file.

It may be good to note that even though the latter case is not defined by the standards, every implementation of awk on every OS I know does read the unterminated last line and adds the newline at the end of the file, so IMO this behavior cannot be labeled as a GNU extension.
# 11  
Old 10-16-2018
Quote:
Originally Posted by apmcd47
Assuming you are using Unix/Linux and are only interested in appending a line terminator to the last line if it doesn't already exist, you could use ed:
Code:
$ echo wq | ed myfile
Newline appended
18
18
 $


Andrew
Like awk and sed, ed has unspecified behavior if the file you are editing is not a text file. The code above may work on some systems, but it is not portable and will not work on all UNIX/Linux/BSD systems!

However, if you change ed to ex, the above code should work portably.
This User Gave Thanks to Don Cragun For This Post:
# 12  
Old 10-16-2018
Quote:
Originally Posted by Scrutinizer
Hi Don, I see where you are going, I may have read the question wrong, I interpreted it as "add a carriage return is it is missing at the end of a line (which ends in linefeed)". So for that, my suggestion does not use extensions - GNU or otherwise - to the standard. I did not mean to also correct a missing newline at the end of the file.

It may be good to note that even though the latter case is not defined by the standards, every implementation of awk on every OS I know does read the unterminated last line and adds the newline at the end of the file, so IMO this behavior cannot be labeled as a GNU extension.
Hi Scrutinizer,
Note that the title of this thread (and the description in post #1 in this thread) says "add Carriage Return and Line Feed"; not "add Carriage Return before Line Feed". The description to me sounded like the input file is a DOS text file with <carriage-return><line-feed>(AKA <newline>) line separators that the submitter wants to turn into a DOS text file with complete DOS lines including a <carriage-return><line-feed> terminator at the end of the last line.

The standards say that the behavior is only defined for awk if all input files are text files. It doesn't make any exception to that requirement for files that are text files except for a partial final line. (The standards do make that exception for the ex utility.)

I am almost positive that PWB UNIX, UNIX System III, UNIX System V, and the Solaris /usr/xpg4/bin versions of awk (at least through early Solaris 10 updates) dropped partial lines without feeding them (with or without an added <newline>) through the script. But, I don't have access to any of those systems to verify it as this point. BSD/macOS awk does add a <newline> terminator to a partial line at the end of a file so it is an extension that is provided by GNU and other systems.
# 13  
Old 10-16-2018
Quote:
Originally Posted by Don Cragun
Hi Scrutinizer,
Note that the title of this thread (and the description in post #1 in this thread) says "add Carriage Return and Line Feed"; not "add Carriage Return before Line Feed".
I have been on this forum long enough that I do not expect exact specifications in the title. I interpreted this to mean a CRLF line ending instead of a LF ending. Do nothing when there is CRLF and change when only LF. It did not make sense to me otherwise.
Quote:
The description to me sounded like the input file is a DOS text file with <carriage-return><line-feed>(AKA <newline>) line separators that the submitter wants to turn into a DOS text file with complete DOS lines including a <carriage-return><line-feed> terminator at the end of the last line.

The standards say that the behavior is only defined for awk if all input files are text files. It doesn't make any exception to that requirement for files that are text files except for a partial final line. (The standards do make that exception for the ex utility.)
I know, I already said that I did not try to correct a failing newline at the end of the file with my suggestion. That is not how I interpreted it, I may be wrong..
Quote:
I am almost positive that PWB UNIX, UNIX System III, UNIX System V, and the Solaris /usr/xpg4/bin versions of awk (at least through early Solaris 10 updates) dropped partial lines without feeding them (with or without an added <newline>) through the script. But, I don't have access to any of those systems to verify it as this point. BSD/macOS awk does add a <newline> terminator to a partial line at the end of a file so it is an extension that is provided by GNU and other systems.
I don't know about really ancient versions, but I just tested with oawk, nawk, and /usr/xpg4/bin/awk on Solaris 10u11, awk on HPUX and AIX and Tru64. All read the unterminated last line and add a newline.. It may be that on older versions of Solaris it did not do that, but I think the awk utility has not changed in this respect (why would they put effort in "fixing" oawk or nawk if they are only there for legacy reasons?).

Last edited by Scrutinizer; 10-16-2018 at 10:15 PM..
# 14  
Old 10-19-2018
Hi Don,

One small change , what if I only need to add the LF if it does not exist at the end of the file , do I just remove the \r from the code ?

Please advise
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Losing carriage return (X0D) after running awk command

Hi Forum. I'm running the following awk command to extract the suffix value (pos 38) from the "AM00" record and append to the end of the "AM01" record. awk 'substr($0,13,4)=="AM00" {SUFFIX = substr($0,38,2)} substr($0,13,4)=="AM01" {$0 = $0 SUFFIX} 1' before.txt > after.txt Before.txt:... (2 Replies)
Discussion started by: pchang
2 Replies

2. Shell Programming and Scripting

line carriage return characters

Hi, I would like to insert the line carriage retrun characters on each line. (2 Replies)
Discussion started by: koti_rama
2 Replies

3. Shell Programming and Scripting

refine awk command in replacing carriage return

Hi, need your help in below,I have 4 types of file need to be processed so that it will replace carriage return in Remarks column with <:::> Remarks column position may varies in different types of file. sample file: col1|col2|col3|col4|col5|col6|col7|Remarks|col9|col10... (8 Replies)
Discussion started by: agathaeleanor
8 Replies

4. Shell Programming and Scripting

Why sed command deletes last line in a file if no carriage return?

Hi I am using sed command to make SCORE=somevalue to SCORE=blank in a file. Please see the attached lastline.txt file. After executing the below command on the file, it removes the last line. cat lastline.txt | sed 's/SCORE=.*$/SCORE=/g' > newfile.txt Why does sed command remove the... (3 Replies)
Discussion started by: ashok.k
3 Replies

5. Programming

Carriage return or line feed issues

I keep running into the same problem with the following script. Every time it prints the carrage (line feed) char when I test. I believe that the issue is in the group by but I do not see it. The code is as follows. SET FEED OFF SET ECHO OFF SET HEADING OFF SET LINESIZE 1000 SET PAGESIZE... (1 Reply)
Discussion started by: sherrod6970
1 Replies

6. Shell Programming and Scripting

Insert a line including Variable & Carriage Return / sed command as Variable

I want to instert Category:XXXXX into the 2. line something like this should work, but I have somewhere the wrong sytanx. something with the linebreak goes wrong: sed "2i\\${n}Category:$cat\n" Sample: Titel Blahh Blahh abllk sdhsd sjdhf Blahh Blah Blahh Blahh Should look like... (2 Replies)
Discussion started by: lowmaster
2 Replies

7. UNIX for Dummies Questions & Answers

To remove carriage return between the line

Hi, I have a situation where I need to remove the carriage return between the lines. For.eg. The input file: 1,ad,"adc sdfd",edf 2,asd,"def fde",asd The output file should be 1,ad,adc sdfd,edf 2,asd,def fde,asd Thanks Shash (5 Replies)
Discussion started by: shash
5 Replies

8. Shell Programming and Scripting

How to insert carriage return before line feed?

I am doing some edi where translations had to be run on unix. Generally when I run the translations on windows, the output file has both carriage returns and line feed where as when ran on unix will have only line feed. I need to insert carriage return before the line feed. Is there some tool... (2 Replies)
Discussion started by: huey ing
2 Replies

9. Shell Programming and Scripting

Removing Carriage Return and or line feed from a file

Hello I'm trying to write a shell script which can remove a carriage return and/or line feed from a file, so the resulting file all ends up on one line. So, I begin with a file like this text in file!<CR> line two!<CR> line three!<CR> END!<CR> And I want to end up with a file... (1 Reply)
Discussion started by: tbone231
1 Replies

10. Shell Programming and Scripting

carriage return/line feeds

Hello, I have a file that has got carriage returns in it and I want to take them out. Anyone know how I can do this in a ksh? thanks (4 Replies)
Discussion started by: pitstop
4 Replies
Login or Register to Ask a Question