Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove characters other than ISO8859-1

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove characters other than ISO8859-1
# 1  
Old 09-22-2010
Remove characters other than ISO8859-1
Hi please help in writing a script for replacing all the non-iso8859-1 characters to question marks.

I need a pattern of this kind
"sed s/<non-iso char range>/?/g < ipfile > opfile"

Please help me in this.
# 2  
Old 09-22-2010
My first solution to this would be to look at a table of the iso8859-1 encoding, decide which byte values are undesirable in the situation's context, and then construct some byte ranges using tr and octal escape sequences.
# 3  
Old 09-22-2010
See if you have the iconv utility program installed.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove first 2 characters and last two characters of each line

here's what im trying to do. i have a file containing lines similar to this: data.txt: 1hsRmRsbHRiSFZNTTA1dlEyMWFkbU5wUW5CSlIyeDFTVU5SYjJOSFRuWmpia0ZuWXpKV2FHTnRU 1lKUnpWMldrZFZaMG95V25oYQpSelEyWTBka2QyRklhSHBrUjA1b1kwUkJkd3BOVXpWM1lVaG5k... (5 Replies)
Discussion started by: SkySmart
5 Replies

2. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies

3. Shell Programming and Scripting

Remove the last 9 characters of a filename

Hi All! Please can someone help, I have a dir with the following files: ~-rw-r--r-- 1 emmuser users 2087361 Oct 16 15:50 MPGGSN02_20131007234519_24291.20131007 -rw-r--r-- 1 emmuser users 2086837 Oct 16 15:50 MPGGSN02_20131007233529_24272.20131007 -rw-r--r-- 1 emmuser ... (7 Replies)
Discussion started by: fretagi
7 Replies

4. Shell Programming and Scripting

Remove characters from the file

i know , the below question has been repeated. can you guys guide me . I have the below input 999999 xxxxxxxxxxxxxx 123.45 2013-05-02 08:14 1 1 1 xxxx 999999 xxxxxxxxxxxxxx 123.45 2013-06-02 02:14 1 4 1 dddd i need to remove from the column 54 to 70 , as like the below output.... (9 Replies)
Discussion started by: expert
9 Replies

5. Shell Programming and Scripting

Remove characters from text

I have a file which looks like this. I only show first 11 lines of the file followed by some text that appears at the end of every file. 1. file:///path1/path2/path3/path4/251192.dat (score 3.849384, docid 142923) 2. file:///path1/path2/path3/path4/173859.dat (score 3.831033, docid 75365) 3.... (4 Replies)
Discussion started by: shoaibjameel123
4 Replies

6. UNIX for Dummies Questions & Answers

How do I remove ^M characters with VI

I have a file with all kinds of ^M at the end of each line. How the heck can these be removed? I tried a global search and replace, but it doesn't seem to work. Thanks! (8 Replies)
Discussion started by: HmmBerger
8 Replies

7. Shell Programming and Scripting

how to remove characters from a string

Hi. for the following line: Var1=${Array} now Array has text as "{hello there}" how do I remove the {} brackets before assigning the string to Var1? Thanks. (3 Replies)
Discussion started by: shadow_boi
3 Replies

8. UNIX for Advanced & Expert Users

remove characters

hi i have a file with these strings: 123_abc_X1116990 how to get rid of 123_abc_ and keep only X1116990? I have columns of these: 123_abc_X1134640 123_dfg_X1100237 123_tyu_X1103112 123_tyui_X1116990 thx (5 Replies)
Discussion started by: melanie_pfefer
5 Replies

9. Solaris

en_US.ISO8859-1 Table

Hy together, I doesn't find a table of en_US.IS08859-1. Have someone a link or same else? Thanks Urs (1 Reply)
Discussion started by: MuellerUrs
1 Replies

10. UNIX for Dummies Questions & Answers

How to remove Characters before '~'

Hi, I am having a file which contains records as follows: DETAIL_KEY~12344|ACTIVE_PASSIVE~Y|AVG_SIZE_OF_RESPONSE~123123131 DETAIL_KEY~12344|ACTIVE_PASSIVE~Y|AVG_SIZE_OF_RESPONSE~123123131 DETAIL_KEY~12344|ACTIVE_PASSIVE~Y|AVG_SIZE_OF_RESPONSE~123123131... (4 Replies)
Discussion started by: Amey Joshi
4 Replies
Login or Register to Ask a Question