Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Removing Embedded Newline from Delimited File

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing Embedded Newline from Delimited File
# 1  
Old 12-18-2008
Removing Embedded Newline from Delimited File
Hey there - a bit of background on what I'm trying to accomplish, first off. I am trying to load the data from a pipe delimited file into a database. The loading tool that I use cannot handle embedded newline characters within a field, so I need to scrub them out.

Solutions that I have tried so far:

1) From a thread here in 2005:
{record = record $0
if (gsub(/"/,"&", record) % 2 )
{ record = record " "
next
}
}
{
print record
record = ""
}

Problems - This worked beautifully on the test data, Then, it was working just fine on the main data... until I received the following error:
"The result [...] of the gsub function cannot be longer than 3000 bytes.

Did I mention that the field with embedded newline characters is going to be loaded as a character large object into the database? Granted, it's only going to be about 6k at max, but that's still more than gsub can handle.

Another note - the test data didn't have any embedded double-quotes. I doubt that this would cause a problem, but in the interest of full disclosure, I should state it.

2) Monster regex:

sed -n '
H
g
s/\n//g
h
/^"\(\(""\)*[^"]*\)*"\(;"\(\(""\)*[^"]*\)*"\)*$/{p;s/.*//g;h;d;}
$p
' filename

Problem - this removes EVERY newline from the script, not just the in-line ones. Definitely can't use this to load the data. Plus, it take a pretty substantial chunk of CPU to run through it.

Other issues with the data:
The "good" newlines at the end of each record are in the same format as the embedded newlines. The FTP client that they use must auto-apply dos2unix when it detects a "text" filetype.

Any help with this would be appreciated. If you need me to clear anything up, let me know.

-Brandon
# 2  
Old 12-18-2008
Not very clear.
Could you paste a sample input record and expected output record for the same.
# 3  
Old 12-18-2008
Sure.

Input:

Code:
PDK12415|11/11/2008|PDK-01724|RandomBlah|Yes|Yes|Yes|No|No|No|09/12/2006|09/04/2008|"Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
"||
PDK28124|11/12/2008|PDK-02315|RandomBlarg [...]

Desired Results:

Code:
PDK12415|11/11/2008|PDK-01724|RandomBlah|Yes|Yes|Yes|No|No|No|09/12/2006|09/04/2008|"Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah."||
PDK28124|11/12/2008|PDK-02315|RandomBlarg [...]

Obviously I'm not loading "blah" into the database, but you get the idea.
Last edited by radoulov; 12-18-2008 at 11:43 AM.. Reason: added code tags
# 4  
Old 12-18-2008
Try
Code:
grep -v ^$ infile| sed -e :a -e 'N; s/\n//; ta'| sed 's/||P/||\nP/g'

# 5  
Old 12-18-2008
Code:
grep -v ^$ infile| sed -e :a -e 'N; s/\n//; ta'| sed 's/||P/||\nP/g'

Thanks, but if the first character of a new line won't always start with P, would I change the P to a [A-z]?

Also, I just realized something about the sample record that I included. It assumes that the last column won't have any data in it. For the file that I have now, that is true, but it's not a guarantee. So, let's adjust the input and desired results.

Input:

Code:
 
PDK12415|11/11/2008|PDK-01724|RandomBlah|Yes|Yes|Yes|No|No|No|09/12/2006|09/04/2008|"Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
 
Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah.
"|stuff|
QWB28124|11/12/2008|PDK-02315|RandomBlarg [...]

Desired results:

Code:
 
PDK12415|11/11/2008|PDK-01724|RandomBlah|Yes|Yes|Yes|No|No|No|09/12/2006|09/04/2008|"Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah blah blah blah. "Blah blah", or "blah blah blah blah", blah blahs the blah."|stuff|
QWB28124|11/12/2008|PDK-02315|RandomBlarg [...]

# 6  
Old 12-18-2008
Something like this?

Code:
perl -0777 -pe'
  tr/\n//d;
  s/((?:.*?\|){14})/$1\n/g
  ' infile

# 7  
Old 12-18-2008
Perfect!

Now, it's become apparent that I have to learn how to craft (or at the very least read and understand) regular expressions, so that I can do similar things without any issue.

I get the "translate all newlines to null" part of the perl script, and I can read that the second line is going to replace ... something... with itself and a newline through the entire script. Now it's just a point of translating ((?:.*?\|){14}). I should be able to handle that though.

Thanks again for the help,
Brandon
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates from delimited file based on 2 columns

Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker... Column #1 is a simple ID, which is used to identify the duplicate. Once dups are identified, I need to only keep the one... (2 Replies)
Discussion started by: kevinprood
2 Replies

2. Shell Programming and Scripting

Script for removing newline character from file

Hi below is my file. cat input.dat 101,abhilash,1000 102,prave en,2000 103,partha,4 000 10 4,naresh,5000 (its just a example file) and my output should be: 101,abhilash,1000 102,praveen,2000 103,partha,4000 104,naresh,5000 below is my code cat input.dat |tr -d '\n' >... (6 Replies)
Discussion started by: abhilash_nakka
6 Replies

3. Shell Programming and Scripting

Removing dupes within 2 delimited areas in a large dictionary file

Hello, I have a very large dictionary file which is in text format and which contains a large number of sub-sections. Each sub-section starts with the following header : #DATA #VALID 1 and ends with a footer as shown below #END The data between the Header and the Footer consists of... (6 Replies)
Discussion started by: gimley
6 Replies

4. UNIX for Dummies Questions & Answers

Removing empty lines at the end of a Tab-delimited file

I'm trying to remove all of the empty lines at the end of a Tab delimited file. They have no data just tabs. I've tried may things, here are a couple: sed /^\t.\t/d File1 > File2 sed /^\t{44}/d File1 > File2 What am I missing? (9 Replies)
Discussion started by: SirHenry1
9 Replies

5. Shell Programming and Scripting

Help with removing embedded linefeeds

Greetings all, i have csv file with pipe separated columns SSN|NAME|ADDRESS|FILLER 123|abc|myaddress|xxx 234|BBB|my add ress broken up|yyy In the example above, the second record is broken into multiple lines. I need to keep going until I find a "|" since this issue is with the... (14 Replies)
Discussion started by: stayalive
14 Replies

6. Shell Programming and Scripting

Removing ^M and the newline that follows it.

Hi Gurus, Apologies as I feel like this must be answered already on here somewhere but I just can't find it. I find many people looking to remove all \n and \r (CR and LF) or one or the other but the only times I've found someone trying to remove them only when both are together they've found... (7 Replies)
Discussion started by: Leedor
7 Replies

7. Shell Programming and Scripting

Read Embedded Newline characters with read (builtin) in KSH93

Hi Guys, Happy New Year to you all! I have a requirement to read an embedded new-line using KSH's read builtin. Here is what I am trying to do: run_sql "select guestid, address, email from guest" | while read id addr email do ## Biz logic goes here done I can take care of any... (6 Replies)
Discussion started by: a_programmer
6 Replies

8. Shell Programming and Scripting

Removing blanks in a text tab delimited file

Hi Experts I am very new to perl and need to make a script using perl. I would like to remove blanks in a text tab delimited file in in a specfic column range ( colum 21 to column 43) sample input and output shown below : Input: 117 102 650 652 654 656 117 93 95... (3 Replies)
Discussion started by: Faisal Riaz
3 Replies

9. UNIX for Advanced & Expert Users

Issue with Removing Carriage Return (^M) in delimited file

Hi - I tried to remove ^M in a delimited file using "tr -d "\r" and "sed 's/^M//g'", but it does not work quite well. While the ^M is removed, the format of the record is still cut in half, like a,b, c c,d,e The delimited file is generated using sh script by outputing a SQL query result to... (7 Replies)
Discussion started by: sirahc
7 Replies

10. Shell Programming and Scripting

Help needed in removing intermediate segments from a pipe delimited segment file

Hi, I just stuckup in doing some regular expressions on a file. I have data which has multiple FHS and BTS segments like: FHS|12121|LOCAL|2323 MSH|10101|POTAMAS|2323 PID|121221|THOMAS|DAVID|23432 OBX|2342|H1211|3232 BTS|0000|MERSTO|LIABLE FHS|12121|LOCAL|2323 MSH|10101|POTAMAS|2323... (3 Replies)
Discussion started by: naren_0101bits
3 Replies
Login or Register to Ask a Question