Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Problem with output awk and sed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Problem with output awk and sed
# 8  
Old 01-24-2014
Where did you get the file? I don't think awk is 'inventing' the garbage characters, I think they are in the original file.
# 9  
Old 01-24-2014
Original file have no such character. when i am opening a file with vi, everything is ok, but after awk operation i am getting problem, Please advise me is it some problem with my environment variable or what, i am totally confuse.
# 10  
Old 01-24-2014
it would take an incredibly sick, faulty version of awk, or bad memory errors, for awk to create all those garbage characters out of nowhere.

Code:
hexdump -C inputfile

# 11  
Old 01-24-2014
Quote:
Originally Posted by Corona688
it would take an incredibly sick, faulty version of awk, or bad memory errors, for awk to create all those garbage characters out of nowhere.

Code:
hexdump -C inputfile

It is showing file like, how to get my file in original condition.

Code:
00027f80  77 00 77 00 77 00 2e 00  71 00 61 00 74 00 61 00  |w.w.w...r.a.t.p.|

---------- Post updated at 12:13 PM ---------- Previous update was at 12:11 PM ----------

Quote:
Originally Posted by learnbash
It is showing file like, how to get my file in original condition.

Code:
00027f80  77 00 77 00 77 00 2e 00  71 00 61 00 74 00 61 00  |w.w.w...r.a.t.p.|


I applied awk command on another server output is same. I am using Fedora 17 at my desktop, is it a problem of my desktop?
# 12  
Old 01-24-2014
^@ is a null. The original file is 16 bits per character. When viewed on an 8 bits per character system the nulls do not show up. I had a file like that once. My file had been created on Windows.
# 13  
Old 01-24-2014
Your file is not ASCII at all but 16-bit Unicode, and as I suspected (but you refused to say) must have come from a Windows system. I guess your version of vi either detected and converted it, or stripped out all the nulls before displaying...

Use iconv to convert it into something your UNIX utilities can understand.

Code:
iconv -f UTF16LE -t UTF8 < inputfile | awk ...

Last edited by Corona688; 01-24-2014 at 01:34 PM.. Reason: iconv, not iconf
This User Gave Thanks to Corona688 For This Post:
learnbash
# 14  
Old 01-24-2014
Quote:
Originally Posted by Corona688
Your file is not ASCII at all but 16-bit Unicode, and as I suspected (but you refused to say) must have come from a Windows system. I guess your version of vi either detected and converted it, or stripped out all the nulls before displaying...

Use iconv to convert it into something your UNIX utilities can understand.

Code:
iconf -f UTF16LE -t UTF8 < inputfile | awk ...

Oh i am sorry, it is generated by some windows software, between how can i verify that file is having 16-bit unicode character? How to check this format in linux?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problem in formatting output in sed / awk

I have a file like this : ! 1 ! 542255 ! 50,140.00 ! ! 2 ! 551717 ! 5,805.00 ! ! 3 ! 551763 ! 8,130.00 ! ! 4 ! 551779 ! 750.00 ! ! 5 ! 551810 ! 56,580.00 ! ! 6 ! 551816 ! 1,350.00 ! ! 7 ! 551876 ! 360.00 ! ! 8 ! 551898 ! ... (10 Replies)
Discussion started by: adam1969in
10 Replies

2. Shell Programming and Scripting

Output on one line using awk or sed

I have a file of 100,000 lines in the below format: answer.bed chr1 957570 957852 NOC2L chr1 976034 976270 PERM1 chr1 976542 976787 PERM1 I need to get each on one line and so far what I have tried doesn't seem to be working. Thank you... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. Shell Programming and Scripting

Use of awk/sed to filter out fdisk output

Hi , I am trying to filter out the below output of fdisk -l command : fdisk -l Disk /dev/sda: 42.9 GB, 42949672960 bytes 255 heads, 63 sectors/track, 5221 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 ... (9 Replies)
Discussion started by: omkar.jadhav
9 Replies

4. Shell Programming and Scripting

Pipe awk's output to sed for deletion

Hi Friends, I am using a command that prints certain lines from a file. For ex: cat input abc chr1 456 def chr1 789 ghi chr1 999 jjj chr1 777 jhk chr7 914 My command awk '{if($2=="chr1" && $3>=456 && $3<=999) {print $0}}' OFS="\t" input Output being printed is abc chr1 456 (7 Replies)
Discussion started by: jacobs.smith
7 Replies

5. Shell Programming and Scripting

need awk or sed help to reformat output

We have the following output: server1_J00_data_20120711122243 server1_J00_igs_20120711122243 server1_J00_j2ee_20120711122243 server1_J00_sec_20120711122243 server1_J00_data_20120711131819 server1_J00_igs_20120711131819 server1_J00_j2ee_20120711131819 server2_J00_data_20120711122245... (10 Replies)
Discussion started by: ux4me
10 Replies

6. UNIX for Dummies Questions & Answers

Appending columns at the end of output using awk/sed

Hi , I have the below ouput, =====gopi===== assasassaa adsadsadsdsada asdsadsadasdsa sadasdsadsd =====kannan=== asdasdasd sadasddsaasd adasdd =====hbk=== asasasssa .... .. I want the output like as below, not able paste here correctly. (2 Replies)
Discussion started by: eeegopikannan
2 Replies

7. UNIX for Advanced & Expert Users

Using awk or sed need the output in the new file

Please find the input file as given below: 2012/02/29 11:00:00~~CRITICAL~For customer 00000476 no daily files were found in the 010137933 account directory. 2012/02/29 11:00:00~~CRITICAL~For customer 05006802 no daily files were found in the 010115166 account directory. 2012/02/29... (0 Replies)
Discussion started by: av_sagar
0 Replies

8. UNIX for Dummies Questions & Answers

awk output problem

Hi Gurus, I have a file which has some fields separated with multiple spaces or single space. data 1 2 3 4 5 6 4 5 5 7 7 8 9 4 6 10 and so on..... The problem I am facing is the output of the awk program... (2 Replies)
Discussion started by: sanjay.login
2 Replies

9. Shell Programming and Scripting

Awk & sed query for output

Hello, I have a file. its content are like below. mdn:87439842 imsi:23082038203 Ctime:12082010 01:20:10 mdn:9324783783 imsi:402349823322 Ctime: 12072010 01:20:10 mdn:87439842 imsi:23082038203 Ctime: 23072010 01:20:10 mdn:87439842 imsi:23082038203 Ctime:18072010 01:20:10 mdn:87439842... (3 Replies)
Discussion started by: Sanket11
3 Replies

10. UNIX for Dummies Questions & Answers

using awk or sed to print output from one file

dear i have one file regarding >abshabja>sdksjbs>sknakna>snajxcls so i want to be output like >abshabja >sjkabjb >sknakna >snajxcls Any using awk or sed will help thanks (2 Replies)
Discussion started by: cdfd123
2 Replies
Login or Register to Ask a Question