Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: File conversion and removing special characters from a file in Linux
Operating Systems Linux File conversion and removing special characters from a file in Linux Post 302863485 by dhruuv369 on Monday 14th of October 2013 11:22:23 AM
Old 10-14-2013
File conversion and removing special characters from a file in Linux
I have a .CSV file when I check for the special characters in the file using the command
Code:
cat -vet filename.csv

, i get very lengthy lines with "^@", "^I^@" and "^@^M" characters in between each alphabet in all of the records. Using the code below
Code:
file filename.csv

I get the output as
Quote:
filename.csv: Little-endian UTF-16 Unicode English character data, with very long lines, with CRLF, CR line terminators
I have a script to remove the control M (^M) from the file, whose output returns me an error saying : cannot execute binary file.

I know that ^I represent a tab. I also have a script to convert ^I to comma delimited file but Can anyone help me format the file with respect to the error and also ^@.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing special characters in file

I have file special.txt with the following data. <header info> 123$ty5%98&0asd 1@356fgbv78 09*&^5jkns43( ...........some more rows. In my output file, I want to eliminate all the special characters in my file and I want all other data. need some help. (6 Replies)
Discussion started by: srivsn
6 Replies

2. Shell Programming and Scripting

removing special characters @ EOL

How to remove special chracters @ END OF EACH LINE in a file file1.txt: 0003073413^M 0003073351^M 0003073379^M 0003282724^M 0003323334^M 0003217159^M 0003102760^M 0002228911^M I used the below command but it is not working ? perl -pi -e 's/^M\/g' file1.txt (6 Replies)
Discussion started by: ali560045
6 Replies

3. AIX

Removing a filename which has special characters passed from a pipe with xargs

Hi, On AIX 5200-07-00 I have a find command as following to delete files from a certain location that are more than 7 days old. I am being told that I cannot use -exec option to delete files from these directories. Having said that I am more curious to know how this can be done. an sample... (3 Replies)
Discussion started by: jerardfjay
3 Replies

4. Solaris

removing special characters, white spaces from a field in a file

what my code is doing, it is executing a sql file and the resullset of the query is getting stored in the text file in a fixed format. for that fixed format i have used the following code:: Code: awk -F":"... (2 Replies)
Discussion started by: priyanka3006
2 Replies

5. Shell Programming and Scripting

Removing special characters

Dear Friends, I want to remove text between two patters. Problem is, it has random special characters like \ / | * ` ~ ! $ etc. These random special characters has no fixed length. But these special characters are appearing between a fixed pattern e.g. DM&^%#|#!\/?CT Expected output... (14 Replies)
Discussion started by: anushree.a
14 Replies

6. Shell Programming and Scripting

Removing Special Character from File.

Hi, My file has this special character "^M" I would like to remove this characters. eg: abc,abc,^M i tried using sed but doesnt work. i used octal dump command to see special character it returns following: 015 \r Appreciate your reply. (6 Replies)
Discussion started by: pinnacle
6 Replies

7. UNIX for Dummies Questions & Answers

awk for removing special characters and extra commas

Hi, I have a .csv file which as empty lines with comma and some special characters in 3rd column as below. Source data 1,2,3,4,%#,6 ,,,,,, 1,2,3,4,5,6 Target Data 1,2,3,4,5,6I need to remove blank lines and special charcters I am trying to get this using the below awk awk -F","... (2 Replies)
Discussion started by: shruthidwh
2 Replies

8. Shell Programming and Scripting

Removing special characters - Control M

I have developed a small script to remove the Control M characters that get embedded when we move any file from Windows to Unix. For some reason, its not working in all scenarios. Some times I still see the ^M not being removed. Is there anything missing in the script: cd ${inputDir}... (7 Replies)
Discussion started by: vskr72
7 Replies

9. Shell Programming and Scripting

Removing blank/white spaces and special characters

Hello All , 1. I am trying to do a task where I need to remove Blank spaces from my file , I am usingawk '{$1=$1}{print}' file>file1Input :- ;05/12/1990 ;31/03/2014 ; Output:- ;05/12/1990 ;31/03/2014 ;This command is not removing all spaces from... (6 Replies)
Discussion started by: himanshu sood
6 Replies

10. UNIX for Beginners Questions & Answers

Replacing string/special characters using a 'conversion' table

Hi, Does anyone know if there is a script or program available out there that uses a conversion table to replace special characters from a file? I am trying to remove some special characters from a file but there are several unprintable/control characters that some I need to remove but some I... (2 Replies)
Discussion started by: newbie_01
2 Replies

LEARN ABOUT SUNOS

iconv_ko

iconv_ko(5)							File Formats Manual						       iconv_ko(5)

NAME

       iconv_ko - code set conversions in Korean (ko) locale

AVAILABILITY

       SUNWkleu

DESCRIPTION

       The following code set conversions are supported:

		       Code Set Conversions Supported
       Code	       Symbol	       TargetCode      Symbol

       Wansung	       ko_KR-euc       Johap	       ko_KR-johap92
       Wansung	       ko_KR-euc       Packed	       ko_KR-johap
       Wansung	       ko_KR-euc       N-Byte	       ko_KR-nbyte
       Wansung	       ko_KR-euc       ISO-2022-KR     ko_KR-iso2022-7
       Johap	       ko_KR-johap92   Wansung	       ko_KR-euc
       Packed	       ko_KR-johap     Wansung	       ko_KR-euc
       N-Byte	       ko_KR-nbyte     Wansung	       ko_KR-euc
       ISO-2022-KR     ko_KR-iso2022-7 Wansung	       ko_KR-euc

       The conversions are performed as like following descriptions.

Wansung to Johap and Johap to Wansung
       For  the  conversion  between  Wansung  which is in Korean EUC representation of KS C 5601-1992 and also known as Completion code and Johap
       which is described at KS C 5601-1992 Annex 3: Supplementary Code System (2 Byte Johap Code System) and also known as Combination code,  all
       characters  can	be  mapped  if	conversion is from Wansung to Johap.  However, from Johap to Wansung conversion, certain Hangul characters
       which are not in KS C 5601-1992 precompsed character set will not be able to mapped. Such characters will be mapped into underscore charac-
       ter  (octal  value  of 137) of KS C 5636.  In case of Hanja and special symbol characters, they will follow the mapping rules of Table 3 of
       Annex 3 of KS C 5601-1992.

Wansung to Packed and Packed to Wansung
       For the conversion between Wansung and Packed which is described at KS C 5601-1987 Annex 3: Supplementary Code System (2  Byte  Johap  Code
       System),  all  characters except Hanja and special symbol characters can be mapped if conversion is from Wansung to Packed.  Such Hanja and
       special symbol characters will be mapped into underscore character (octal value 137) of KS C 5636.  From Packed to Wansung conversion, cer-
       tain  Hangul  chracters will not be able to mapped since not every Hangul Packed code is representable in Wansung.  Such characters will be
       mapped into underscore character of KS C 5636.

Wansung to N-Byte and N-Byte to Wansung
       For the conversion between Wansung and N-Byte which is described at KS C 5601-1992 Annex 4: 7 Bit Hangul  Alphabet  codes,  all	characters
       except  Hanja and special symbol characters can be mapped.  Such Hanja and special symbol characters will be mapped into underscore charac-
       ter (octal value 137) of KS C 5636.

Wansung to ISO-2022-KR and ISO-2022-KR to Wansung
       For the conversion between Wansung and ISO-2022-KR which is described at RFC 1557: Korean Character Encoding  for  Internet  Messages,  all
       characters can be mapped.

FILES

       /usr/lib/iconv/ko_KR-euc%ko_KR-johap92.so
					       Shared object for Korean EUC to Johap code conversion
       /usr/lib/iconv/ko_KR-euc%ko_KR-johap.so Shared object for Korean EUC to Packed code conversion
       /usr/lib/iconv/ko_KR-euc%ko_KR-nbyte.so Shared object for Korean EUC to N-Byte code conversion
       /usr/lib/iconv/ko_KR-euc%ko_KR-iso2022-7.so
					       Shared object for Korean EUC to ISO-2022-KR code conversion
       /usr/lib/iconv/ko_KR-johap92%ko_KR-euc.so
					       Shared object for Johap to Korean EUC code conversion
       /usr/lib/iconv/ko_KR-johap%ko_KR-euc.so Shared object for Packed to Korean EUC code conversion
       /usr/lib/iconv/ko_KR-nbyte%ko_KR-euc.so Shared object for Johap to Korean EUC code conversion
       /usr/lib/iconv/ko_KR-iso2022-7%ko_KR-euc.so
					       Shared object for ISO-2022-KR to Korean EUC code conversion

SEE ALSO

       iconv(1), iconv(3), iconv(5)

								    10 Mar 1995 						       iconv_ko(5)
All times are GMT -4. The time now is 09:11 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy