We have a huge file which has just one really large line; about 500 MB. I want to
1. Count all the occurrences of a phrase
2. Replace the phrase with another.
Trying to open it using vi has not helped as it complains that it is too large. Can any script help? Please advise.
Can you show a sample of this file? Maybe it just uses a different separator than newline, in which case awk could process it via changing the value of RS and ORS...
i've not dealt with a file of a single 500MB line, so I play and create one:
while i cat onto itself until it reached >500MB.
now we play:
well, seems there isn't many tools that aren't line based.. awk will use a single character separator, unless it's gawk.
well 256 ram and 256 swap. OP has issues opening the file in vi so I figured I'd run the tests in my low-end VPS.
as Corona pointed out, there must be a different record separator that could be used, so that a program doesn't attempt to load the entire line into memory.
more information is needed about the format of the file. i'm unaware of a standard unix visual editor that'd properly open such a file (though i'm very inexperienced in the subject, i can usually hack away solutions but i've not enough information)
Well, at least the problem with "vi" i can explain:
A file cannot be edited in the same place where it is stored. This is the reason why i.e.
works, while
will not (the same is true for awk, etc.).
Some (GNU-)versions of sed circumvent this principal limitation by introducing inline-editing ("-i"), which works the same way interactive editors like vi do: they create a copy at program start and only upon saving/finishing the work they copy this over the original file.
vi typically uses /var/tmp per default, but can be configured to use other places too (at least to my knowledge all versions of vi offer such an option via the .exrc file). If this filesystem has not enough free space to hold the copy an attempt to edit the file will fail even if there would be enough space in memory to hold it.
Another limitation is the maximum line length: this is a system limitation and how long lines can grow is laid down in the kernel header file limits.h in the constant "LINE_MAX".
Hi.
Pardon me if I'm posting a duplicate thread but..
I have a text file with over 150 Million records, file size is in the range if MB(close to GB).
The requirement is to read ALL the lines excepting the FIRST LINE which is the file header and the LAST LINE which is it's trailer record.
... (8 Replies)
Hi ,
I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options
Regards
Raj (1 Reply)
Hi friends,
Issue1:
I have a text file with the first line like this
#chrom start end Readcount_A Normalized_Readcount_A ReadcountB Normalized_Readcount_B Fc_A_vs_B pvalue_A_vs_B FDR_A_vs_B Fc_B_vs_A pvalue_B_vs_A FDR_B_vs_A <a href="http://unix.com/">Link</a>
How can I change it to the... (11 Replies)
Hi Experts,
I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Hello All,
My file content is:
DROP TABLE
"FACT_WORLD";
CREATE TABLE "FACT_WORLD" (
"AR_ID" INTEGER NOT NULL,
"ORG_ID" INTEGER NOT NULL
)
DATA CAPTURE NONE
COMPRESS YES;
I want to change this file to have entries in one... (6 Replies)
I have file which contains around 5000 lines.
The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters.
I want to delete the lines
a> if it starts with 1 and if 576th postion is a digit i,e 0-9
or
b> if it starts with 0 or 9(i,e header and footer)
... (4 Replies)
Hi,
How can I edit a line in a file?
For example, a.txt contains:
start: 1 2 3 4
stop: a b c d
and I want to change "3" to "9"
and to add "5" after "4"
the result should be (a.txt):
start: 1 9 3 4 5
stop: a b c d
Thanks,
zed (5 Replies)
i need help..!!!!
i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows.
on OS SUN Solaris version 8
How i can remove first line of the text file.
Please suggest me for solutions.
Thank you very much in advance:) (5 Replies)
I am trying to edit each line in a file. The file has several columns delimitted by '|'.
I need to take out the last two columns.
Each line starts with a unique word through which I am storing the lines in a variable and cutting the last two colums.
But, when I am echoing the line, it is... (2 Replies)
Hello,
I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it?
Any help would be... (3 Replies)