Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Removing inline binary data from txt file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing inline binary data from txt file
# 1  
Old 04-04-2013
Removing inline binary data from txt file
I am trying to parse a file but the filehas binary data inline mixed with text fields.
I tried the binutils strings function , it get the binary data out but put the char following the binary data in a new line .

input file
Code:
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(–QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(Ì11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(ûPAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R

I need to get to
Code:
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012() deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016() deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012() deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R

Just want to remove the binary data , the brackets surrounding the binary data are not material

any suggestions

This is breaking what I am trying to do here
My earlier q on parsing :
https://www.unix.com/shell-programmin...ll-script.html
# 2  
Old 04-04-2013
try:
Code:
perl -pe 's/[^[:ascii:]]//g' infile

This User Gave Thanks to rdrtx1 For This Post:
tasmac
# 3  
Old 04-04-2013
There is some other binary data which is still causing this to file.
When copying to this editor it causes a line break

Code:
app_id:1996 pgm_num:0 branch:TBNY ord_num:0440010(
Q6I) deal_number:0

---------- Post updated at 12:11 PM ---------- Previous update was at 11:33 AM ----------

These are probabaly control chars ( not the ususal ^M ) ones .
Is there a way to remove all control characters from a file

tr ?
# 4  
Old 04-04-2013
Try this gawk code:
Code:
gawk '{gsub(/[[:cntrl:]]/,x)}1' inputfile > outputfile

# 5  
Old 04-04-2013
nope
Code:
sandbox]$>nawk '{gsub(/[[:cntrl:]]/,x)}1' bin4lines.txt
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(Ì11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(ûPAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R
Q6I) deal_number:0 m:0 branch:NY ord_num:0440000010(
/sandbox]$>

# 6  
Old 04-04-2013
This should remove all non-ASCII characters:
Code:
tr -d '\200-\377' < inputfile > outputfile

outputfile cannot be the same as inputfile.
This User Gave Thanks to Corona688 For This Post:
tasmac
# 7  
Old 04-04-2013
I don't think nawk supports that character class.

Use /usr/xpg4/bin/awk instead:
Code:
/usr/xpg4/bin/awk '{gsub(/[[:cntrl:]]/,x)}1' bin4lines.txt
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

2. Windows & DOS: Issues & Discussions

Xterm logging on Cygwin/X - binary data in log file.

I have Cygwin/X installed on Windows 7. In an xterm, I turned on logging via Main Options > Log to File. When I open my log file with Vim I get a warning that it might be binary. Looking through the file I see what I think are VT datastream escape characters. It makes it hard to use the... (1 Reply)
Discussion started by: gctaylor
1 Replies

3. Shell Programming and Scripting

Data extraction from .txt file

Hey all, i´ve got the following problem: i´m aquiring data with an instrument and i get data in a .txt file. This is how the txt file looks like: Report of AU program poptau F1P=-49.986ppm F2P=-110.014ppm Target directory for serfile: D:/data/Spect500/nmr/Thoma/882 Linear... (17 Replies)
Discussion started by: expikx
17 Replies

4. Shell Programming and Scripting

To log binary file output to a txt file

Hi, I wrote a small script whose function is to execute the postemsg provided if the threshold breaches. I want to log this postemsg messages to a log file. But I am not able to do. Can someone throw some light on how to log the output of this. I am pasting a snippet of that code. ... (2 Replies)
Discussion started by: dbashyam
2 Replies

5. Programming

writing binary/struct data to file

I am trying to write binary data to a file. My program below: #include <stdlib.h> #include <stdio.h> struct tinner { int j; int k; }; struct touter { int i; struct tinner *inner; }; int main() { struct touter data; data.i = 10; struct tinner... (4 Replies)
Discussion started by: radiatejava
4 Replies

6. Shell Programming and Scripting

how to check the file data type(ascii or binary)

hi i am receiving a file from one system , i have to verify the format of the file data i.e whether the data is in acii format or binary format, please help thanks in advance satya (1 Reply)
Discussion started by: Satyak
1 Replies

7. Shell Programming and Scripting

Script for removing text from a txt file

Hello, So I wanted to write a very simple script to remove some information from a text file and save it as something else. For example I have a text file (let's call it txt) with three rows of numbers: 0 0 1 9 8 7 5 0 6 7 9 0 0 7 9 8 1 1 6 4 0 6 0 0 9 8 4 6 0 9 2 8 1 And I want to... (2 Replies)
Discussion started by: hertingm
2 Replies

8. UNIX for Dummies Questions & Answers

Binary txt file received when i use uuencode to send txt file as attachment

Hi, I have already read a lot of posts on sending attachments in unix...but none of them were of help for my problem...so here goes.. i wanna attach a text file and send to a mail id..used the following code : uuencode "$File1" "$File1" ;|mail -s "$Mail_sub" abc@abc.com it works... (2 Replies)
Discussion started by: ash22
2 Replies

9. UNIX for Advanced & Expert Users

get data from .txt file

Hi I have a file diskspace.txt the data in that file is asFilesystem kbytes used avail %used Mounted on /dev/vg06/lvol1 18870272 12099836 6673336 64% /dist know i am writing a script to get the above bold data and that data need to be used as a input. any idea... (4 Replies)
Discussion started by: gkrishnag
4 Replies

10. UNIX for Dummies Questions & Answers

Binary data to text file conversion

Dear Sir; i want to know how the binary data convert to text file or readablw format (ASCII).If possible pl. help me for the software and where it is available for download. i.e. (1 Reply)
Discussion started by: auro123
1 Replies
Login or Register to Ask a Question