Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Formatting large Ascii file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Formatting large Ascii file
# 1  
Old 10-24-2008
Formatting large Ascii file
Hi, I've got a very large ASCII file (1.5 GB), which doesn't have line terminators, making it impossible to read it right, i need to format it like:
add line terminators.
delete some lines on the file.
delete some fields on all the remainings lines (can be identified by a string).

After getting the file under the correct format i will inject it into a mysql DB, file details:

ASCII text, with very long lines, with no line terminators, i already tried tr and sed , but i cannot get it right.

hexdump head:

0000000 3136 3937 3934 3134 2032 2020 2020 4120
0000010 5249 2020 4d54 2053 4120 5249 2020 3137
0000020 3131 3131 2030 2020 2020 2020 2020 2020
0000030 2020 2020 2020 2020 2020 2020 2020 2020
*
0000070 2020 2020 2020 2020 2020 2020 3032 3730
0000080 3231 3732 3031 3131 3934 3032 3730 3231
0000090 3732 3330 3030 3030 3130 3038 3130 2020
00000a0 3236 3137 3630 3638 2030 2020 2020 4120
00000b0 5249 2020 4d54 2053 4120 5249 2020 3137
00000c0 3131 3131 2030 2020 2020 2020 2020 2020
00000d0 2020 2020 2020 2020 2020 2020 2020 2020
*
0000110 2020 2020 2020 2020 2020 2020 3032 3730
0000120 3231 3732 3031 3131 3934 3032 3730 3231
0000130 3732 3231 3030 3030 3130 3038 3130 2020
0000140 3236 3537 3332 3737 2036 2020 2020 4120
0000150 5249 2020 4d54 2053 4120 5249 2020 3137
0000160 3131 3131 2030 2020 2020 2020 2020 2020
0000170 2020 2020 2020 2020 2020 2020 2020 2020
*

--

Best regards.
# 2  
Old 10-24-2008
You will have to write a dd command or maybe C or perl, something like that. Most Unix tools have a limit to line size.


How is the format supposed to be? We need examples of input and expected output.
# 3  
Old 10-24-2008
examples.
Thanks for your reply, Example input, keep in mind it doesn't have line terminators , so when you open it on big terminals it get imposible to read:

617955555 AIR TMS AIR 7111110 2007122710114920071227030000018001 627166666 AI
R TMS AIR 7111110 2007122710114920071227120000018001 627577777 AIR TMS AIR 7111
110 2007122710114920071227120000018001 627588888 AIR AME AIR 7311110
2007122710114920071227030000018001 627599999 AIR AME AIR 7311110

example output:

617955555 AIR TMS AIR
627166666 AIR TMS AIR
627577777 AIR TMS AIR
627588888 AIR AME AIR
627599999 AIR AME AIR

Best regards.
# 4  
Old 10-24-2008
dd is your friend.
Start with this:
Code:
dd ifile=inputfilename ofile=outputfilename cbs=22 conv=unblock

It is late for me - play around with the 22 block size until it works. I think your input block size is 22 with byte 22=space. Read the dd manpage.
# 5  
Old 10-26-2008
Done!
cbs=20 did the magic, then sed '/^6/ !d' , best regards.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Convert EBCDIC(.DAT) FILE into ASCII FILE

Hi Team, I am having 100 EBCDIC files (i.e. DAT extension) and need to convert them into ASCII File by unix shell script. I tried with DD Command but its not providing output as expected. Sample Text: ------------------ גהאבדוחס@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Expected Output:... (2 Replies)
Discussion started by: JSM
2 Replies

2. Shell Programming and Scripting

Convert Hex to Ascii in a Ascii file

Hi All, I have an ascii file in which few columns are having hex values which i need to convert into ascii. Kindly suggest me what command can be used in unix shell scripting? Thanks in Advance (2 Replies)
Discussion started by: HemaV
2 Replies

3. UNIX for Dummies Questions & Answers

After Ftp'ing file to destination how to check the file if it is in correct ASCII and not corrupted

Hi Folks, While transferring file from FTP software like Filezilla the files gets corrupted. Is there any way I can check if the recently transferred file is in ASCII and not corrupted. I have tried using file -i filename command which does tell if the file character set is ASCII or binary... (6 Replies)
Discussion started by: Khan28
6 Replies

4. Red Hat

Unable to convert EBCDIC file to ASCII file

Hello all, To give you all a little bit of background. We recently migrated from HP-UX to Redhat Linux and one of the command I used to run on HP-UX to convert an EBCDIC file to ASCII file isn't working on Linux. The code is as follow: cat workout2.dat | dd cbs=250 conv=block conv=ascii... (3 Replies)
Discussion started by: sethmj
3 Replies

5. UNIX for Advanced & Expert Users

Large volume file formatting

Hi, I have a file which is around 193 gb in size. This file has tonnes of spaces and I need to sanitize it. I tried to use awk script to split this file but it gave me an error like line to long... As of now I am using a sed command to search replace the spaces; however its too slow for such a... (2 Replies)
Discussion started by: darshanw
2 Replies

6. Shell Programming and Scripting

ASCII Text formatting

Hi All, I have a requirement to create a formatted text. That is., I have a plain text composed to send it through an Email using mailx. I need to change the formatting of the text like BOLD, ITALIC, UNDERINED, COLOURED, INDENT etc., First of all, is it possible in UNIX? If yes, please let... (3 Replies)
Discussion started by: mohan_kumarcs
3 Replies

7. Shell Programming and Scripting

convert ascii values into ascii characters

Hi gurus, I have a file in unix with ascii values. I need to convert all the ascii values in the file to ascii characters. File contains nearly 20000 records with ascii values. (10 Replies)
Discussion started by: sandeeppvk
10 Replies

8. Shell Programming and Scripting

How to convert English text file to ASCII File?

file abc abc: English text I want to convert the above into file abc file: ascii text (1 Reply)
Discussion started by: laknar
1 Replies

9. UNIX Desktop Questions & Answers

ASCII file

I've been having trouble trying to read an ASCII file. I'm on an IRIX machine, by the way. I've tried "cat" and I get a bunch of unreadable text, and the "string" command gets the "Command not Found" error. Please advise. Thanks. (1 Reply)
Discussion started by: sherbet808
1 Replies

10. UNIX for Dummies Questions & Answers

How can I ... (Modifying large ASCII files)

Hi Everybody! Situation: I have a large ASCII file (for example: 1-2 Mbytes) without linebreaks (\n). Task: I like inserting linebreaks after all 420 digits (byte). (pattern: *\n*\n*\n...etc.) My problem: How? :-) I like using shell script or (maybe) AWK (short) program. Please,... (2 Replies)
Discussion started by: hviktor
2 Replies
Login or Register to Ask a Question