Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Conditional replacement of a delimiter

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Conditional replacement of a delimiter
# 1  
Old 04-19-2011
Conditional replacement of a delimiter
Hello,

I'm new to this forum but this seems like the place to ask this question.

I have a pipe delimited data file with the fields except for the header being encased in double quotes. I found out that some of the fields have an trash pipe within the data itself. I'd like to conditionally remove any pipe that is not a field delimiter.

Here is some sample data
Code:
Email|DateCreated|FirstName|LastName|PostCode|Address_line1|Address_line2|City
"sample@email.com"|"2007/11/12 17:57:04"|"Keith A| Glass"|""|"20110"|""|""|""
"sample@verizo|n.net"|"2007/11/12 18:18:56"|"ESPN"|"Mi||er"|"06/25/1955"|""|""|"NY"
"sample@ao|.com"|"2007/11/12 18:20:25"|"Mary"|"Shelley"|"05/16/1845"|"James St.||Apt. 1"|""|"CT"
"samp|e@diagnostics.com"|"2007/11/12 18:20:28"|"Jam|es"|"Sample"|""|"110 Perkins Street||Apt. 6"|""|""

As you can see the header has the pipe delimited, and those are fine. But those pipes that are within the field are shifting the data when I load into the database.

Thank you for your help.
Samah
Last edited by Scott; 04-19-2011 at 10:09 PM.. Reason: Added code tags
# 2  
Old 04-19-2011
if all your valid fields are quoted, than you could do this:
Code:
sed 's/"|"/","/g' data | sed '2,$ s/|//g'

The first sed will change all delimiting pipes into commas (all char-triplets "|" into triplets ",", then all pipes you are left with are the messy ones. Piping it again to sed to get rid of them (or you could replace them with some other character, if you wished). Second sed command operates onlines 2 until end-of-file, to keep the header intact.
# 3  
Old 04-20-2011
Quote:
Originally Posted by samahs
I'd like to conditionally remove any pipe that is not a field delimiter
Try this:
Code:
awk -F\" '{for(i=2;i<NF;i+=2)gsub("\|",x,$i)}1' OFS=\" file

# 4  
Old 04-20-2011
Or try..
Code:
sed 's/"\([^|"]\+\)|\+\([^|"]\+\)"/"\1\2"/g' inputfile > outfile

# 5  
Old 04-26-2011
Thank you everyone for the help. All of those worked very nicely.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Conditional replacement in CSV files

Hello, I have many CSV files with variable number of rows and columns. Sample of few problematic CSV files. ,,Price,Price,Price,Price,Price,Price,Price,Price,Price,Qty Date,Sl,AAA,BBB,CCC,DDD,EEE,FFF,GGG,HHH,PriQueue,%busy 30/07/2014,1,AAA,BBB,CCC,DDD,EEE,FFF,GGG,HHH,NA,0... (8 Replies)
Discussion started by: reddyr
8 Replies

2. Shell Programming and Scripting

Conditional replacement of columns in a text file

Hello scriping expert friends, I have 2 requirements on replacing fields of text files: I have lot of data with contents like below: Requirement-1: The digit after 0 should always be changed to 1 (3 Replies)
Discussion started by: magnus29
3 Replies

3. Shell Programming and Scripting

Perl Code to change file delimiter (passed as argument) to bar delimiter

Hi, Extremely new to Perl scripting, but need a quick fix without using TEXT::CSV I need to read in a file, pass any delimiter as an argument, and convert it to bar delimited on the output. In addition, enclose fields within double quotes in case of any embedded delimiters. Any help would... (2 Replies)
Discussion started by: JPB1977
2 Replies

4. Shell Programming and Scripting

Shell script to put delimiter for a no delimiter variable length text file

Hi, I have a No Delimiter variable length text file with following schema - Column Name Data length Firstname 5 Lastname 5 age 3 phoneno1 10 phoneno2 10 phoneno3 10 sample data - ... (16 Replies)
Discussion started by: Gaurav Martha
16 Replies

5. Shell Programming and Scripting

selective replacement of delimiter

I have a file with two fields seperated by comma data looks like below with the header The o/p should look like this Basically, the req is to replace only the first occuring comma with pipe can we do this with any commands (2 Replies)
Discussion started by: dsravan
2 Replies

6. Shell Programming and Scripting

conditional replacement

Hi all, I need a bash, sed, awk script or one liner to do the following task: This is the format of a text file: 2010-06-11 20:01 902656 HOP-W-100412-1.doc 2010-11-05 18:01 364447 NEX-W-101104-1 2010-07-06 10:01 64512 Cerintele 2010-07-06 10:01 599420 content 2010-07-19 14:01 1785344... (7 Replies)
Discussion started by: supervazi
7 Replies

7. Shell Programming and Scripting

Conditional tab replacement sed/awk

Hi I am struggling to find a solutions to this problem: I have a directory full of files and I wish to: read each line of each file and if any one line in those files is longer than 72 characters I want to replace any tab characters with a space character. Ive been... (3 Replies)
Discussion started by: benackland
3 Replies

8. Shell Programming and Scripting

HELP Need in SED/PERL conditional line replacement

Hi , I need some help on perl/sed conditional replacement The situation is like below . I have a file contents like below . AAA|BBB|CCC|DDD AAA|BCF|CCC|HHH AAA|BVF|JJJ|KKK Here in the above file . I know my second column value (taking "|" as my delimited ) Basically I have to... (3 Replies)
Discussion started by: robin.r888
3 Replies

9. Shell Programming and Scripting

Substring based on delimiter, finding last delimiter

Hi, I have a string like ABC.123.XYZ-A1-B2-P1-C4. I want to delimit the string based on "-" and then get result as only two strings. One with string till last hyphen and other with value after last hyphen... For this case, it would be something like first string as "ABC.123.XYZ-A1-B2-P1" and... (6 Replies)
Discussion started by: gupt_ash
6 Replies

10. Shell Programming and Scripting

Replacement of Delimiter

Dear all, i have a proble. in my input file i have records with delimiter like aa-------bb------cc--vghjav---ef----kjd dj--------ih------yy--ujdjkkl---dd----jid now i want to replace the delimiter "-" with "~" i have used a command i.e cat FILENAME | tr "-" "~" >> Newfile this command... (3 Replies)
Discussion started by: panknil
3 Replies
Login or Register to Ask a Question