Sponsored Content
Top Forums Shell Programming and Scripting Sed - remove special characters Post 302700885 by nakaedu on Friday 14th of September 2012 11:18:18 AM
Old 09-14-2012
Quote:
Originally Posted by Chubler_XL
Can you paste the output of head -1 file1 | od -c for us so we can see the exact byte codes you have in this file.
Code:
 
 
This is what I got 

$ head -1 lista1.csv | od -c
0000000 357 273 277 \ \ n a k a m a \ c o m p
0000020 a r t \ ; 9 4 . 2 6 0 . 9 7 4
0000040 B y t e s ; 7 ; 1 \r \n
0000055

---------- Post updated at 11:16 AM ---------- Previous update was at 11:16 AM ----------

Quote:
Originally Posted by RudiC
In principle not too difficult:
Code:
$ sed 's/^´╗┐//' file1

, but you should be aware that these character graphics chars usually belong to a multibyte character set like utf-8 or so which may impose restrictions.
Thanks, but I'm afraid this didn't work

---------- Post updated at 11:18 AM ---------- Previous update was at 11:16 AM ----------

Quote:
Originally Posted by Scrutinizer
Or to remove multibyte characters, you could try:
Code:
LANG=C tr -d '[\200-\377]' < infile

That has worked perfectly,

Thank you very much.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

remove special and unicode characters

Hi, How do I remove the lines where special characters or Unicode characters appear? The following query does work but I wonder if there is a better way. cat test.txt | egrep -v '\)|#|,|&|-|\(|\\|\/|\.' The following lines show that my query is incomplete. Warning: The word "*Khan" is... (1 Reply)
Discussion started by: shantanuo
1 Replies

2. Shell Programming and Scripting

Remove special characters from string

Hi there, I'd like to write a script that removes any set of character from any string. The first argument would be the string, the second argument would be the characters to remove. For example: $ myscript "My name's Santiago. What's yours?" "atu" My nme's Snigo. Wh's yors? I wrote the... (11 Replies)
Discussion started by: chebarbudo
11 Replies

3. Shell Programming and Scripting

How to remove special characters from each line?

Hello, Is there a simpler way to remove special characters (color codes) from each lines in a log file? I use sed like in the example below but I think there should be a more simple way to achieve the same result: $ cat -vet file1 ^, , , , Maybe to convert the file somehow? ... (5 Replies)
Discussion started by: majormark
5 Replies

4. UNIX for Dummies Questions & Answers

How to Remove Special Characters

Dear Members, We have a file which contains some special characters. I need to replace these special character by a new line character(\n). The Special character is \x85. I am not sure what this character means and how we can remove it. Any inputs are greatly appreciated. Thanks... (5 Replies)
Discussion started by: sandeep_1105
5 Replies

5. Shell Programming and Scripting

remove special characters

hello all I am writing a perl code and i wish to remove the special characters for text. I wish to remove all extended ascii characters. If the list of special characters is huge, how can i do this using substitute command s/specialcharacters/null/g I really want to code like... (3 Replies)
Discussion started by: vasuarjula
3 Replies

6. Shell Programming and Scripting

Remove string between two special characters

Hi All, I have a variable like AVAIL="\ BACK:bkpstg:testdb3.iad.expertcity.com:backtest|\ #AUTH:authstg:testdb3.iad.expertcity.com:authiapd|\ TEST:authstg:testdb3.iad.expertcity.com:authiapd|\ " What I want to do here is that If a find # before any entry, remove the entire string... (5 Replies)
Discussion started by: engineermayur
5 Replies

7. Shell Programming and Scripting

sed or tr to remove specific group of special characters

Hi, I have a input of the form: ..., word1, word2, word3... I want out put of the form word1, word2, word3 I tried echo '..., word1, word2, word3...' | tr -d '...,' but that takes out the commas in the middle too so I get word1 word2 word3 but I want the commas in the middle. ... (3 Replies)
Discussion started by: forumbaba
3 Replies

8. Shell Programming and Scripting

How to remove some special characters in a string?

Hi, I have string like this ="Lookup Procedure" But i want the output like this Lookup Procedure =," should be removed. Please suggest me the solution. Regards, Madhuri (2 Replies)
Discussion started by: srimadhuri
2 Replies

9. Shell Programming and Scripting

How to remove special characters?

Hi Gurus, I have file which contains some unicode charachator like "ü". I want to replace it with some charactors. I searched in internet and got command sed "s/ü/-/g", but I don't know how to type ü in unix command line. Please help me for this one. Thanks in advance (7 Replies)
Discussion started by: ken6503
7 Replies

10. Shell Programming and Scripting

Remove Special Characters Within Text

Hi, I have a "|" delimited file that is exported from a database. There is one column in the file which has description/comments entered by some application user. It has "Control-M" character and "New Line" character in between the text. Hence, when i export the data, this record with the new... (4 Replies)
Discussion started by: tarun.trehan
4 Replies
TR(1)							    BSD General Commands Manual 						     TR(1)

NAME
tr -- translate characters SYNOPSIS
tr [-cs] string1 string2 tr [-c] -d string1 tr [-c] -s string1 tr [-c] -ds string1 string2 DESCRIPTION
The tr utility copies the standard input to the standard output with substitution or deletion of selected characters. The following options are available: -c Complements the set of characters in string1, that is -c ab includes every character except for 'a' and 'b'. -d The -d option causes characters to be deleted from the input. -s The -s option squeezes multiple occurrences of the characters listed in the last operand (either string1 or string2) in the input into a single instance of the character. This occurs after all deletion and translation is completed. In the first synopsis form, the characters in string1 are translated into the characters in string2 where the first character in string1 is translated into the first character in string2 and so on. If string1 is longer than string2, the last character found in string2 is dupli- cated until string1 is exhausted. In the second synopsis form, the characters in string1 are deleted from the input. In the third synopsis form, the characters in string1 are compressed as described for the -s option. In the fourth synopsis form, the characters in string1 are deleted from the input, and the characters in string2 are compressed as described for the -s option. The following conventions can be used in string1 and string2 to specify sets of characters: character Any character not described by one of the following conventions represents itself. octal A backslash followed by 1, 2 or 3 octal digits represents a character with that encoded value. To follow an octal sequence with a digit as a character, left zero-pad the octal sequence to the full 3 octal digits. character A backslash followed by certain special characters maps to special values. a <alert character>  <backspace> f <form-feed> <newline> <carriage return> <tab> v <vertical tab> A backslash followed by any other character maps to that character. c-c Represents the range of characters between the range endpoints, inclusively. [:class:] Represents all characters belonging to the defined character class. Class names are: alnum <alphanumeric characters> alpha <alphabetic characters> blank <blank characters> cntrl <control characters> digit <numeric characters> graph <graphic characters> lower <lower-case alphabetic characters> print <printable characters> punct <punctuation characters> space <space characters> upper <upper-case characters> xdigit <hexadecimal characters> With the exception of the ``upper'' and ``lower'' classes, characters in the classes are in unspecified order. In the ``upper'' and ``lower'' classes, characters are entered in ascending order. For specific information as to which ASCII characters are included in these classes, see ctype(3) and related manual pages. [=equiv=] Represents all characters or collating (sorting) elements belonging to the same equivalence class as equiv. If there is a sec- ondary ordering within the equivalence class, the characters are ordered in ascending sequence. Otherwise, they are ordered after their encoded values. An example of an equivalence class might be ``c'' and ``ch'' in Spanish; English has no equivalence classes. [#*n] Represents n repeated occurrences of the character represented by #. This expression is only valid when it occurs in string2. If n is omitted or is zero, it is interpreted as large enough to extend string2 sequence to the length of string1. If n has a lead- ing zero, it is interpreted as an octal value, otherwise, it's interpreted as a decimal value. EXIT STATUS
tr exits 0 on success, and >0 if an error occurs. EXAMPLES
The following examples are shown as given to the shell: Create a list of the words in file1, one per line, where a word is taken to be a maximal string of letters: tr -cs "[:alpha:]" " " < file1 Translate the contents of file1 to upper-case: tr "[:lower:]" "[:upper:]" < file1 Strip out non-printable characters from file1: tr -cd "[:print:]" < file1 COMPATIBILITY
AT&T System V UNIX has historically implemented character ranges using the syntax ``[c-c]'' instead of the ``c-c'' used by historic BSD implementations and standardized by POSIX. AT&T System V UNIX shell scripts should work under this implementation as long as the range is intended to map in another range, i.e. the command tr [a-z] [A-Z] will work as it will map the '[' character in string1 to the '[' character in string2. However, if the shell script is deleting or squeezing characters as in the command tr -d [a-z] the characters '[' and ']' will be included in the deletion or compression list which would not have happened under an historic AT&T System V UNIX implementation. Additionally, any scripts that depended on the sequence ``a-z'' to represent the three characters 'a', '-', and 'z' will have to be rewritten as ``a-z''. The tr utility has historically not permitted the manipulation of NUL bytes in its input and, additionally, stripped NUL's from its input stream. This implementation has removed this behavior as a bug. The tr utility has historically been extremely forgiving of syntax errors, for example, the -c and -s options were ignored unless two strings were specified. This implementation will not permit illegal syntax. STANDARDS
The tr utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. It should be noted that the feature wherein the last character of string2 is duplicated if string2 has less characters than string1 is permitted by POSIX but is not required. Shell scripts attempting to be portable to other POSIX systems should use the ``[#*]'' convention instead of relying on this behavior. BUGS
tr was originally designed to work with US-ASCII. Its use with character sets that do not share all the properties of US-ASCII, e.g., a sym- metric set of upper and lower case characters that can be algorithmically converted one to the other, may yield unpredictable results. tr should be internationalized. BSD
March 23, 2004 BSD
All times are GMT -4. The time now is 08:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy