New Line in CSV file Post: 302967564

Sponsored Content

Top Forums Shell Programming and Scripting New Line in CSV file Post 302967564 by Don Cragun on Thursday 25th of February 2016 02:02:22 AM

02-25-2016

Registered User

UNIX format text files have a newline character as a line terminator. DOS format text file have a carriage-return character immediately followed by a newline character with the combination of those two characters being the line terminator.

The first file you asked about (INPUT_GIANT_JAN2016.txt) was not a normal DOS format text file. It only had the carriage-return character as the line terminator (except for the last line in the file that had both carriage-return and newline characters). So to fix it, all we had to do was change the carriage-return characters to newline characters. (That left you with a single empty line at the end of the file after tr changed all of the carriage-return characters to newline characters.)

The second file (INPUT_GUARDIAN_JAN2016.txt) is a normal DOS format text file. So instead of changing carriage-return characters to newline characters, we just need to remove the carriage-return characters:

Code:

tr -d '\r' < INPUT_GUARDIAN_JAN2016.txt > NEW_INPUT_GUARDIAN_JAN2016.txt

If you only have files in these two exact formats, you could handle both with:

Code:

tr '\r' '\n' < input_file | grep -v '^$' > fixed_file

This won't work for general text files (which frequently have empty lines separating paragraphs), but it should work for the samples you have shown us so far.

This User Gave Thanks to Don Cragun For This Post:

Don Cragun

View Public Profile for Don Cragun

Find all posts by Don Cragun

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Checking the Empty line in csv file

Hi all, I have one CSV file(MasterFile.csv) consists of two columns. "./incoming/ABC.CSV","./incoming/ABC_CONTROL.txt" "./incoming/PQR.CSV","./incoming/PQR_CONTROL.txt" "./incoming/123.CSV","./incoming/123_CONTROL.txt" I have written a script to read the MasterFile.csv.Here i want to...

2. Shell Programming and Scripting

how to start looping from the second line in .csv file

I have a .csv file and i use the below while loop to navigate through it But i need to loop from the second line since the first line is the header How will i do it?? please help while IFS=, read Filename Path size readonly do echo "Filename -> ${Filename}" echo "Path -> ${Path}" echo...

3. Shell Programming and Scripting

How to remove line break in a csv file

Hi Experts, My requirement is to read the csv file and need to remove if any line break in it. sample data: Row1: "Oslo, Symra kino",Oslo,130-7,Symra 1,130-7-91 Row2:"T�nsberg, Brygga Kino SF",T�nsberg,202-1, T�nsberg SF 4,202-1-4 Expected data: Row1: "Oslo, Symra...

4. Shell Programming and Scripting

Read csv file line by line

Folks , i want to read a csv file line by line till the end of file and filter the text in the line and append everything into a variable. csv file format is :- trousers:shirts,price,50 jeans:tshirts,rate,60 pants:blazer,costprice,40 etc i want to read the first line and get...

5. UNIX for Dummies Questions & Answers

Delete first line of a csv file

Hi there How do I delete the first line of a csv file without creating a new file ? Regards, Peter

6. UNIX for Dummies Questions & Answers

skip first line when doing a read of csv file

Folks, how do i skip the first line in a csv, while doing the read of a csv file in to a variable line by line. eg : do echo $line done < $rpt where rpt is path to csv file The initial 1st line is a garbage that i want to avoid, and start reading from 2nd line ...

7. Shell Programming and Scripting

Reading last line of a CSV file

Hi I have a file which I am reading line by line and processing it. But the last line is not getting read in the file loop until I put an enter in the end. #!/bin/ksh -p v_org_id=${P1} FILE=${P2} NEW_FILE_NAME=$APPLPTMP/b1.txt BAKIFS=$IFS IFS=$'\n' exec 0<"$FILE" echo "File to be...

8. Shell Programming and Scripting

Split csv file by line count.

I have a very large csv file that I sort by the data that is in the second column. But what I need to do next is split the file in groups of say around 30,000 lines but don't split the data while there is still like data in the in the second column. Here is some of the data. ...

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk...

10. Shell Programming and Scripting

Column with New Line in CSV file

Hello, Got a CSV file which contains 21 columns Need to convert the file to Pipe delimiter and Few columns text data contains new line Example 1,2,3,"ABC" 8" ABC,5,6,7 1,2,3,"ABC" 8" ABC,5,6,7 ( New Line) 1,2,3,""ABC" 8" ABC", 5,6,7 1,2,3,"ABC" ,5,6,7(New line) Expected...

LEARN ABOUT NETBSD

asa

ASA(1)							    BSD General Commands Manual 						    ASA(1)

NAME

     asa -- interpret carriage-control characters

SYNOPSIS

     asa [file ...]

DESCRIPTION

     The asa utility reads files sequentially, mapping FORTRAN carriage-control characters to line-printer control sequences, and writes them to
     the standard output.

     The first character of each line is interpreted as a carriage-control character.  The following characters are interpreted as follows:

     <space>  Output the rest of the line without change.

     0	      Output a <newline> character before printing the rest of the line.

     1	      Output a <formfeed> character before printing the rest of the line.

     +	      The trailing <newline> of the previous line is replaced by a <carriage-return> before printing the rest of the line.

     Lines beginning with characters other than the above are treated as if they begin with <space>.

EXIT STATUS

     The asa utility exit 0 on success, and >0 if an error occurs.

EXAMPLES

     To view a file containing the output of a FORTRAN program:
	   asa file

     To format the output of a FORTRAN program and redirect it to a line-printer:
	   a.out | asa | lpr

SEE ALSO

     f77(1)

STANDARDS

     The asa utility conforms to IEEE Std 1003.2-1992 (``POSIX.2'') and to X/Open Commands and Utilities Issue 5 (``XCU5'').

AUTHORS

     J.T. Conklin, Winning Strategies, Inc.

BSD
								September 23, 1993							       BSD