Search and replace from file in awk using a 16 bit text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search and replace from file in awk using a 16 bit text file
# 1  
Old 02-13-2013
Search and replace from file in awk using a 16 bit text file

Hello,
Some time ago a helpful awk file was provided on the forum which I give below:
Code:
NR==FNR{A[NR]=$0;next}{for(j in A){split(A[j],P,"=");for(i=1;i<=NF;i++){if($i==P[1]){$i=P[2]}}}}1

While it works beautifully on English and Latin characters i.e. within the ASCII range of 127, the moment a character beyond 127 is provided for search and replace, the script fails
How do we modify the script to make it work for characters beyond
Code:
A-z

A short sample is provided below for the mapper
Code:
¦=ு
§=ங
¨=ி
©=ீ
ª=்
¬=இ
±=யு
²=வு
´=வூ
µ=பு
·=பூ
¹=யூ
¼=ஈ
¿=நூ
Á=ஹ
Ã=ஊ
Ä=னு
Å=நு
Æ=கு
Ç=ழு
È=ழ
É=ஐ
Ê=து
Ë=மு
Ì=டு
Í=ஆ
Î=சூ
Ï=டீ
Ð=்
Ñ=ணு
Ô=கூ
Õ=ரு

and a sample file for conversion is also provided below:
Code:
îòàõ裪$#
Íîõèæªæ¨áëè¨$#
òôè¨áëªëèóª$#
íÕóª$#
ëè¨áòôè¨.ò$#
îòô¨áèªíëùªõ裪.ò$#
òúªËæëª.âõ$#
ëÕêèáªíëªëèóª.à$#
îòô¨áèª.êè$#
òôè¨áèèª.¬$#
î£èòçª.ö¡ª$#
òæèáõ裪.ô¨$#
¬õèáçªçäª$#A
Íîõèæªæ¨áî£ëª¡ª.ö¡ª$#
õ裪$#
ëè¨á¼òèæª$#C
ëè¨áê衪$#
çèéùáè.Í$#
íËùªõ裪$#
îôëªçè¨á.ô$#
Íîõèæªæ¨áê衪$#
òåªê¨õîòæõäª$#K
î£èò窥çäª$#
ËÕîæòäª$#P
çèæªæ¨áëªâ£áçèùäª$#M
òèë¨ê衪$#
âëÁÕäª$#L
Æ窵òªòèë¨$#S
âçÕëèóª.à$#

Many thanks in advance for the help
# 2  
Old 02-13-2013
I can't see immediately why the script fragment should not work - it works on strings, not on chars unless you have omitted the FS="" assignment.
Pls post/attach input & desired output files, the entire script, and possibly errors that occurred.
# 3  
Old 02-13-2013
Hello,
The mapper is the file in which the conversion rules are given. The character on the left hand side is the input and the one on the right hand side is the output.
The file on which the search and replace operation is to be carried out is given as the input.
I cannot provide output since the script does not work. Basically each 8 bit character should be mapped to a corresponding Unicode character and the output should be Unicode.
I have tried again and I get the same results. It spews out the input file as such.
Many thanks for replying.
# 4  
Old 02-13-2013
These input files do not appear to be in UTF-8 format. Convert them to UTF-8 first.
# 5  
Old 02-13-2013
Dear Scrutinizer,
The mapper file has the following structure:
Code:
Left hand before delimiter: 8 bit. Non Unicode/ upper ASCII in range 135 to 254
Right hand after delimiter. 16 bit Unicode or if so desired could also be UTF-8

The Input file is all in 8 bit Non Unicode i.e. Upper ASCII Range
The aim of the script would be to convert a non-Unicode 8 bit file into Unicode using the search and replace mappings stored in the mapper file.
I hope the explanation is clear.
Many thanks for your query and interest.
# 6  
Old 02-14-2013
hexdump of above mapper file:
Code:
c2 a6 3d e0 af 81  
c2 a7 3d e0 ae 99
c2 a8 3d e0 ae bf

It's definitely not 8 bit on the left hand side (3d = "=" delimiter). Pls provide as attachments (advanced edit) input, mapper, and desired output files.
# 7  
Old 02-14-2013
Many thanks I will try and provide the sample as an attachment and so also the mapper file.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search field in text file and replace value

Hi there, First of all this is my first post here. Thank you in advance for your help. What I am trying to do is the following. I have a text file where each field of each row is separated by a tabulator. Looks like this: ATOM 1 N HSE A 26 3.033 -10.429 -2.262 1.00 17.07 ... (8 Replies)
Discussion started by: doom4
8 Replies

2. Emergency UNIX and Linux Support

Search and replace in text file

Hi, I have gigabytes of text files that I need to search for "&" and replace with "&amp". Is there a way to do this efficiently (like sed command)? Hope you could help. Thanks. (17 Replies)
Discussion started by: daytripper1021
17 Replies

3. Shell Programming and Scripting

Search replace strings between single quotes in a text file

Hi There... I need to serach and replace a strings in a text file. My file has; books.amazon='Let me read' and the output needed is books.amazon=NONFOUND pls if anybody know this can be done in script sed or awk.. i have a list of different strings to be repced by NONFOUND.... (7 Replies)
Discussion started by: Hiano
7 Replies

4. Shell Programming and Scripting

search text file in file if this file contains necessary text (awk,grep)

Hello friends! Help me pls to write correct awk and grep statements for my task: I have got files with name filename.txt It has such structure: Start of file FROM: address@domen.com (12...890) abc DATE: 11/23/2009 on Std SUBJECT: any subject End of file So, I must check, if this file... (4 Replies)
Discussion started by: candyme
4 Replies

5. Shell Programming and Scripting

text file search and replace with awk

hello all greeting for the day i have a text file as the following text.xml abcd<FIELD>123.456</FIELD>efgh i need to replace the value between <FIELD> and </FIELD> by using awk command. please throw some light on this. thank you very very much Erik (5 Replies)
Discussion started by: erikshek
5 Replies

6. Shell Programming and Scripting

search and replace a text in a file

Hi all, I have a requirement where i have to search data between strings 'SELECT' and ';' and replace this text as "SELECT.....;" so that i can export this extracted string into a excel cell. Please suggest on this. (5 Replies)
Discussion started by: goutam_igate
5 Replies

7. UNIX for Dummies Questions & Answers

how can search a String in one text file and replace the whole line in another file

i am very new to UNIX plz help me in this scenario i have two text files as below file1.txt name=Rajakumar. Discipline=Electronics and communication. Designation=software Engineer. file2.txt name=Kannan. Discipline=Mechanical. Designation=CADD Design Engineer. ... (6 Replies)
Discussion started by: kkraja
6 Replies

8. UNIX for Dummies Questions & Answers

search and replace a specific text in text file?

I have a text file with following content (3 lines) filename : output.txt first line:12/12/2008 second line:12/12/2008 third line:Y I would like to know how we can replace 'Y' with 'N' in the 3rd line keeping 1st and 2nd lines same as what it was before. I tried using cat output.txt... (4 Replies)
Discussion started by: santosham
4 Replies

9. Shell Programming and Scripting

How to search and replace text in same file

script is as below v_process_run=5 typeset -i p_cnt=0 pdata=/home/proc_data.log while do # execute script in background dummy_test.sh "a1" "a2" & p_cnt=$p_cnt+1 echo "data : $p_cnt : Y" >> $pdata done file created with following data in... (1 Reply)
Discussion started by: Vrgurav
1 Replies

10. Shell Programming and Scripting

automating file search and replace text

Hi, I am trying something like this: Let's say I have a file called File1 with contents: x=-0.3 y=2.1 z=9.0 I have another file, File2, with contents: xx= yy= zz= (nothing after "="). What I want to do is get the value of x in File1 and set it to xx in File2, i.e., xx=-0.3. And the... (3 Replies)
Discussion started by: ommatidia
3 Replies
Login or Register to Ask a Question