Removing "Hidden Characters" on a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing "Hidden Characters" on a file
# 1  
Old 06-23-2011
Removing "Hidden Characters" on a file

Hi -

I'm having a problem with hidden characters on Linux. When I produced an output from Oracle database, there is a an extra "Hidden Character" included on the output. How can I remove that character? See below:

Quote:
InterUnit$
The extra dollar sign is creating a new line on my .csv output file. I tried using sed, but no luck.

Quote:
sed -e 's/"InterUnit$"/"InterUnit"/g' file.csv > file2.csv
Note: The '$' is not being included on HP-UX machines upon my query.
# 2  
Old 06-23-2011
Hi

Code:
sed -e 's/InterUnit\$/InterUnit/g'  file.csv > file2.csv

Guru
This User Gave Thanks to guruprasadpr For This Post:
# 3  
Old 06-23-2011
Try this..

Code:
 
tr -cd '\11\12\40-\176' < myfile1 > myfile2

This User Gave Thanks to itkamaraj For This Post:
# 4  
Old 06-23-2011
Hi guru,

Thanks for the fast response but my problem is not solved using sed.

I tried using your command but the Linux still cannot interpret the hidden '$' even if we used a escape character. I can only view the '$' using the "set list" function of vi, but I cannot grep the string using a normal vi.

Also, I tried viewing the file using 'od -c' and the output is below:

Quote:
0000560 | I n t e r U n i t \n
The '$' is interpreted as "next line".

---------- Post updated at 03:23 PM ---------- Previous update was at 03:19 PM ----------

Hi itkamaraj,

Can you please explain what the command is doing? I can't see the '$' in there.

UPDATE: I tried the command tr -cd command, but the '$' is still there.
# 5  
Old 06-23-2011
The code i provided is to remove the special characters from the file

Dec value 128 to 225
Hex value 80 to FF

Is the output produced in windows platform ?

try

Code:
 
dos2unix < infile > outf

Code:
 
perl -i -pe 's/\r//g' inputfile

This User Gave Thanks to itkamaraj For This Post:
# 6  
Old 06-23-2011
The files will be generated by the script on Linux box, then send the file as attachment to MS Outlook users.

I tried both of the commands, but the '$' is still there. Am I the only one experiencing this?
# 7  
Old 06-23-2011
just try this.

This will remove all the special characters except the keyboard characters
Code:
perl -i -pe 's/[^\x20-\x7f]//g' examplefile

---------- Post updated at 01:16 PM ---------- Previous update was at 01:07 PM ----------

Is the file generated by perl script ?

did u use chomp method for each variable to remove the new line character ?
This User Gave Thanks to itkamaraj For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies

3. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

4. Shell Programming and Scripting

finding the strings beween 2 characters "/" & "/" in .txt file

Hi all. I have a .txt file that I need to sort it My file is like: 1- 88 chain0 MASTER (FF-TE) FFFF 1962510 /TCK T FD2TQHVTT1 /jtagc/jtag_instreg/updateinstr_reg_1 dff1 (TI,SO) 2- ... (10 Replies)
Discussion started by: Behrouzx77
10 Replies

5. Shell Programming and Scripting

Removing "^M" from the end of a String (i.e. "Ctrl+M")?

Hello All, I have an Expect script that ssh's to a remote server and runs some commands before exiting. One of the commands I run is the "hostname" Command. After I run this command I save the output using this line in the code below... Basically it executes the hostname command, then I... (2 Replies)
Discussion started by: mrm5102
2 Replies

6. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

7. Shell Programming and Scripting

Removing special characeter "~V" in a unix file

I have the Unix XML file as below: <?xml version="1.0" encoding="UTF-8"?> <ReportData version="1.0"><DisplayName>Non-Agency CMO Daily Trade Recap - Hybrids</DisplayName><ReportType>MgmtTradingReport</ReportType><Description>Management Trading... (7 Replies)
Discussion started by: mohsin.quazi
7 Replies

8. Solaris

removing "/" file system from solaris volume

Hi all, I have created a volume for the root device as d0 and the sub mirror for same is d10. the output from metastat d0 is as below I want to clear these volume , as i cant unmount the "/ " file system , please suggest as how can i clear this. Also the required entries are there... (2 Replies)
Discussion started by: kumarmani
2 Replies

9. Shell Programming and Scripting

How to remove "New line characters" and "spaces" at a time

Dear friends, following is the output of a script from which I want to remove spaces and new-line characters. Example:- Line1 abcdefghijklmnopqrstuvwxyz Line2 mnopqrstuvwxyzabcdefghijkl Line3 opqrstuvwxyzabcdefdefg Here in above example, at every starting line there is a “tab” &... (4 Replies)
Discussion started by: anushree.a
4 Replies

10. UNIX for Dummies Questions & Answers

File names that contain "01" act hidden

Haveing an issue. Anytime a file is created with "01" (zero one) in the name - it cannot be viewed by LS or any other file listing command. Although the file is there, it cannot be seen. I can edit it, run it, anything, except see it..... What happened? Any ideas? (8 Replies)
Discussion started by: n9ninchd
8 Replies
Login or Register to Ask a Question