Shell Programming and Scripting

You also have forgotten to tell us what OS you're using. The awk script I suggested will properly handle any number of fields on a line as long as the length of longest line in the output file is no longer than LINE_MAX (typically 2048) bytes. Most of the sed scripts you have been shown in this thread will properly handle files with one or more (but not zero) fields with the same line length limits. Some OSs will be OK with longer lines; others will generate a diagnostic message if a long line is encountered; and, unfortunately, a few others will silently corrupt your data.

If the line length is a problem for your data on your system, we can play games with the awk record separator (instead of field separator) to get around the issue.

So, what is the maximum line length in your input data and how many fields can you have in your output files? (Output line length will be (input line length) + 2 * (number of input fields).)

What is the value of LINE_MAX on your system (output from getconf LINE_MAX)?

What OS are you using (output from uname -a)?

Hey Don

Much thanks for looking into this.I am using AIX 7.2.I dont think length of line is a problem and is very well within limits.I tried the AWk you suggested and I got an error.

A.txt is my pipe delim input file

Code:

cut -d "|" -f 1,2,3,4,5 A.txt > A1.txt 
cut -d "|" -f 6- A.txt > A2.txt

A3.txt is an external pipe delim file

Code:

paste -d "|" A2.txt A3.txt>>A4.txt

A5.txt would be my comma delim output

Code:

awk -F'|' -v OFS='","' 'NF{$1 = $1; $0 = "\"" $0 "\""}1'A4.txt >> A5.txt
 syntax error The source line is 1.
 The error context is
                NF{$1 = $1; $0 = "\"" $0 >>>  "\""}1A4. <<< txt
 awk: Quitting
 The source line is 1.

---------- Post updated at 03:51 AM ---------- Previous update was at 03:42 AM ----------

here is URL for sample of how my input delim file A.txt looks like

http://s24.postimg.org/d8afdzcsl/example.jpg

What if any one of the fields already is double quoted?

Code:

cat file
A|B|C|D
A|B|C|D
A|"B"|C|"D"
A|B|C|D
awk -F'|' -v OFS='","' 'NF{$1 = $1; $0 = "\"" $0 "\""}1' file
"A","B","C","D"
"A","B","C","D"
"A",""B"","C",""D""
"A","B","C","D"

etldev,
There has to be a space between the ' terminating the awk script and the filename being given to awk as an input file:

Code:

awk -F'|' -v OFS='","' 'NF{$1 = $1; $0 = "\"" $0 "\""}1' A4.txt >> A5.txt

RudiC,
The sample input shown in the link provided in post #9 in this thread contains no quotes of any kind (and no commas, either). With the sample data provided (which did not include any empty lines either), the following would be sufficient:

Code:

awk -F'|' -v OFS='","' '{$1 = $1}1' A4.txt >> A5.txt

But I assume that other input files will contain commas. If other input contains any double quote characters, adjustments will be needed and we'll need to know if the double quotes that are present are intended to be literal characters or are quoting other characters in the input. If any of them are intended to be literal double quote characters, we'll also need to know the quoting conventions used by the application that is going to read the CSV file this script is producing.

etldev,
In your sample code you are appending to files A4.txt and A5.txt rather than replacing whatever may have been in those files before. Is that intentional, or did you mean to use > instead of >> in both of those places?

neither of these above AWK codes are giving me desired output as I could still see PIPES as field delims but not commas...

however the awk code which takes care of quotes and comas is just putting commas at the beggining and at the end of the line.

Don

to your point there are no commas within the data now in the input files but there may be situations we might and in those situations the pipes and commas in data have to be selectively converted so that when the ouput is opened in sprdsheet all the data looks intact without any quotes around them (preferrably)

What OS are you using?

I repeat: Why are you using >> A5.txt instead of > A5.txt???

Please put the data you showed us in the JPEG file you posted in a file named A4.txt and run the command:

Code:

awk -F'|' -v OFS='","' 'NF{$1 = $1; $0 = "\"" $0 "\""}1' A4.txt > A5.txt

or, as long as there are no commas in A4.txt, the command:

Code:

awk -F'|' -v OFS=',' '{$1 = $1}1' A4.txt > A5.txt

and then show us the contents of the file A5.txt and any diagnostics produced by the awk command.

Posting a JPEG file doesn't really help with this problem. We need text that we can feed into awk, not pixels that are unintelligible to the UNIX and Linux system text processing utilities.

I have attched the input A4.txt .

When I run the big awk commnd the one which takes care of quotes my output is A5.txt-which is attached but when I run the other awk my output is same as A4.txt