Shell Programming and Scripting

We have a huge file which has just one really large line; about 500 MB. I want to
1. Count all the occurrences of a phrase
2. Replace the phrase with another.

Trying to open it using vi has not helped as it complains that it is too large. Can any script help? Please advise.

Thank you,

To count:

Code:

perl -ne '@x=/phrase/g;print $#x+1 . "\n"' file

To replace:

Code:

perl -i -pe 's/phrase/replace/g' file

Can you show a sample of this file? Maybe it just uses a different separator than newline, in which case awk could process it via changing the value of RS and ORS...

i've not dealt with a file of a single 500MB line, so I play and create one:

Code:

$ cat input
a horse is a horse, of course, of course, and no one can talk to a horse of course. That is, of course, unless the horse is the famous Mr. Ed.

while i cat onto itself until it reached >500MB.
now we play:

Code:

[mute@geek ~]$ perl -ne '@x=/horse/g;print $#x+1 . "\n"' input
Killed
[mute@geek ~]$ grep -o horse input | wc -l
grep: input: Cannot allocate memory
0
[mute@geek ~]$ awk -F 'horse' '{print NF}' input
awk: cmd. line:1: fatal: grow_iop_buffer: iop->buf: can't allocate 536870914 bytes of memory (Cannot allocate memory)
[mute@geek ~]$ gawk -vRS="horse" 'END{print NR}' input
20331009

well, seems there isn't many tools that aren't line based.. awk will use a single character separator, unless it's gawk.

How much physical memory do you have on that machine, neutronscott?

well 256 ram and 256 swap. OP has issues opening the file in vi so I figured I'd run the tests in my low-end VPS.
as Corona pointed out, there must be a different record separator that could be used, so that a program doesn't attempt to load the entire line into memory.
more information is needed about the format of the file. i'm unaware of a standard unix visual editor that'd properly open such a file (though i'm very inexperienced in the subject, i can usually hack away solutions but i've not enough information)

Well, at least the problem with "vi" i can explain:

A file cannot be edited in the same place where it is stored. This is the reason why i.e.

Code:

sed '<something>' infile > outfile

works, while

Code:

sed '<something>' infile > infile

will not (the same is true for awk, etc.).

Some (GNU-)versions of sed circumvent this principal limitation by introducing inline-editing ("-i"), which works the same way interactive editors like vi do: they create a copy at program start and only upon saving/finishing the work they copy this over the original file.

vi typically uses /var/tmp per default, but can be configured to use other places too (at least to my knowledge all versions of vi offer such an option via the .exrc file). If this filesystem has not enough free space to hold the copy an attempt to edit the file will fail even if there would be enough space in memory to hold it.

Another limitation is the maximum line length: this is a system limitation and how long lines can grow is laid down in the kernel header file limits.h in the constant "LINE_MAX".

I hope this helps.