Removing "Hidden Characters" on a file | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Removing "Hidden Characters" on a file

Shell Programming and Scripting


Tags
hidden characters, removing hidden characters

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
Removing "Hidden Characters" on a file

Hi -

I'm having a problem with hidden characters on Linux. When I produced an output from Oracle database, there is a an extra "Hidden Character" included on the output. How can I remove that character? See below:

Quote:
InterUnit$
The extra dollar sign is creating a new line on my .csv output file. I tried using sed, but no luck.

Quote:
sed -e 's/"InterUnit$"/"InterUnit"/g' file.csv > file2.csv
Note: The '$' is not being included on HP-UX machines upon my query.
Sponsored Links
    #2  
Old 06-23-2011
guruprasadpr's Avatar
guruprasadpr guruprasadpr is offline Forum Advisor  
Shrink...ing
 
Join Date: Jun 2009
Last Activity: 21 October 2014, 6:30 AM EDT
Location: India
Posts: 675
Thanks: 30
Thanked 215 Times in 214 Posts
Hi


Code:
sed -e 's/InterUnit\$/InterUnit/g'  file.csv > file2.csv

Guru
The Following User Says Thank You to guruprasadpr For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
    #3  
Old 06-23-2011
itkamaraj's Avatar
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 3 October 2014, 8:50 AM EDT
Posts: 3,059
Thanks: 33
Thanked 659 Times in 638 Posts
Try this..


Code:
 
tr -cd '\11\12\40-\176' < myfile1 > myfile2

The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
    #4  
Old 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
Hi guru,

Thanks for the fast response but my problem is not solved using sed.

I tried using your command but the Linux still cannot interpret the hidden '$' even if we used a escape character. I can only view the '$' using the "set list" function of vi, but I cannot grep the string using a normal vi.

Also, I tried viewing the file using 'od -c' and the output is below:

Quote:
0000560 | I n t e r U n i t \n
The '$' is interpreted as "next line".

---------- Post updated at 03:23 PM ---------- Previous update was at 03:19 PM ----------

Hi itkamaraj,

Can you please explain what the command is doing? I can't see the '$' in there.

UPDATE: I tried the command tr -cd command, but the '$' is still there.
Sponsored Links
    #5  
Old 06-23-2011
itkamaraj's Avatar
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 3 October 2014, 8:50 AM EDT
Posts: 3,059
Thanks: 33
Thanked 659 Times in 638 Posts
The code i provided is to remove the special characters from the file

Dec value 128 to 225
Hex value 80 to FF

Is the output produced in windows platform ?

try


Code:
 
dos2unix < infile > outf


Code:
 
perl -i -pe 's/\r//g' inputfile

The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
    #6  
Old 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
The files will be generated by the script on Linux box, then send the file as attachment to MS Outlook users.

I tried both of the commands, but the '$' is still there. Am I the only one experiencing this?
Sponsored Links
    #7  
Old 06-23-2011
itkamaraj's Avatar
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 3 October 2014, 8:50 AM EDT
Posts: 3,059
Thanks: 33
Thanked 659 Times in 638 Posts
just try this.

This will remove all the special characters except the keyboard characters

Code:
perl -i -pe 's/[^\x20-\x7f]//g' examplefile

---------- Post updated at 01:16 PM ---------- Previous update was at 01:07 PM ----------

Is the file generated by perl script ?

did u use chomp method for each variable to remove the new line character ?
The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
awk command to replace ";" with "|" and ""|" at diferent places in line of file shis100 Shell Programming and Scripting 7 03-16-2011 08:59 AM
Removing special characeter "~V" in a unix file mohsin.quazi Shell Programming and Scripting 7 10-20-2009 05:25 PM
removing "/" file system from solaris volume kumarmani Solaris 2 03-03-2009 03:21 AM
How to remove "New line characters" and "spaces" at a time anushree.a Shell Programming and Scripting 4 09-08-2008 05:55 PM
File names that contain "01" act hidden n9ninchd UNIX for Dummies Questions & Answers 8 06-09-2001 10:25 AM



All times are GMT -4. The time now is 09:50 AM.