Unix/Linux Go Back    


Shell Programming and Scripting Unix shell scripting - KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and shell scripts and shell scripting languages here.

Removing "Hidden Characters" on a file

Shell Programming and Scripting


Tags
hidden characters, removing hidden characters

Closed Linux or Unix Question    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
Removing "Hidden Characters" on a file

Hi -

I'm having a problem with hidden characters on Linux. When I produced an output from Oracle database, there is a an extra "Hidden Character" included on the output. How can I remove that character? See below:

Quote:
InterUnit$
The extra dollar sign is creating a new line on my .csv output file. I tried using sed, but no luck.

Quote:
sed -e 's/"InterUnit$"/"InterUnit"/g' file.csv > file2.csv
Note: The '$' is not being included on HP-UX machines upon my query.
Sponsored Links
    #2  
Old Unix and Linux 06-23-2011
guruprasadpr's Unix or Linux Image
guruprasadpr guruprasadpr is offline Forum Advisor  
Shrink...ing
 
Join Date: Jun 2009
Last Activity: 10 July 2015, 4:39 AM EDT
Location: India
Posts: 676
Thanks: 30
Thanked 215 Times in 214 Posts
Hi


Code:
sed -e 's/InterUnit\$/InterUnit/g'  file.csv > file2.csv

Guru
The Following User Says Thank You to guruprasadpr For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
    #3  
Old Unix and Linux 06-23-2011
itkamaraj's Unix or Linux Image
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 25 June 2015, 3:24 AM EDT
Posts: 3,060
Thanks: 34
Thanked 661 Times in 640 Posts
Try this..


Code:
 
tr -cd '\11\12\40-\176' < myfile1 > myfile2

The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
    #4  
Old Unix and Linux 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
Hi guru,

Thanks for the fast response but my problem is not solved using sed.

I tried using your command but the Linux still cannot interpret the hidden '$' even if we used a escape character. I can only view the '$' using the "set list" function of vi, but I cannot grep the string using a normal vi.

Also, I tried viewing the file using 'od -c' and the output is below:

Quote:
0000560 | I n t e r U n i t \n
The '$' is interpreted as "next line".

---------- Post updated at 03:23 PM ---------- Previous update was at 03:19 PM ----------

Hi itkamaraj,

Can you please explain what the command is doing? I can't see the '$' in there.

UPDATE: I tried the command tr -cd command, but the '$' is still there.
Sponsored Links
    #5  
Old Unix and Linux 06-23-2011
itkamaraj's Unix or Linux Image
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 25 June 2015, 3:24 AM EDT
Posts: 3,060
Thanks: 34
Thanked 661 Times in 640 Posts
The code i provided is to remove the special characters from the file

Dec value 128 to 225
Hex value 80 to FF

Is the output produced in windows platform ?

try


Code:
 
dos2unix < infile > outf


Code:
 
perl -i -pe 's/\r//g' inputfile

The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
    #6  
Old Unix and Linux 06-23-2011
Jin_ Jin_ is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 30 October 2013, 2:48 AM EDT
Posts: 26
Thanks: 16
Thanked 0 Times in 0 Posts
The files will be generated by the script on Linux box, then send the file as attachment to MS Outlook users.

I tried both of the commands, but the '$' is still there. Am I the only one experiencing this?
Sponsored Links
    #7  
Old Unix and Linux 06-23-2011
itkamaraj's Unix or Linux Image
itkamaraj itkamaraj is offline Forum Advisor  
^Kamaraj^
 
Join Date: Apr 2010
Last Activity: 25 June 2015, 3:24 AM EDT
Posts: 3,060
Thanks: 34
Thanked 661 Times in 640 Posts
just try this.

This will remove all the special characters except the keyboard characters

Code:
perl -i -pe 's/[^\x20-\x7f]//g' examplefile

---------- Post updated at 01:16 PM ---------- Previous update was at 01:07 PM ----------

Is the file generated by perl script ?

did u use chomp method for each variable to remove the new line character ?
The Following User Says Thank You to itkamaraj For This Useful Post:
Jin_ (06-23-2011)
Sponsored Links
Closed Linux or Unix Question

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
awk command to replace ";" with "|" and ""|" at diferent places in line of file shis100 Shell Programming and Scripting 7 03-16-2011 08:59 AM
Removing special characeter "~V" in a unix file mohsin.quazi Shell Programming and Scripting 7 10-20-2009 05:25 PM
removing "/" file system from solaris volume kumarmani Solaris 2 03-03-2009 03:21 AM
How to remove "New line characters" and "spaces" at a time anushree.a Shell Programming and Scripting 4 09-08-2008 05:55 PM
File names that contain "01" act hidden n9ninchd UNIX for Dummies Questions & Answers 8 06-09-2001 10:25 AM



All times are GMT -4. The time now is 01:53 PM.