06-13-2012
well 256 ram and 256 swap. OP has issues opening the file in vi so I figured I'd run the tests in my low-end VPS.
as Corona pointed out, there must be a different record separator that could be used, so that a program doesn't attempt to load the entire line into memory.
more information is needed about the format of the file. i'm unaware of a standard unix visual editor that'd properly open such a file (though i'm very inexperienced in the subject, i can usually hack away solutions but i've not enough information)
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hello,
I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it?
Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies
2. UNIX for Dummies Questions & Answers
I am trying to edit each line in a file. The file has several columns delimitted by '|'.
I need to take out the last two columns.
Each line starts with a unique word through which I am storing the lines in a variable and cutting the last two colums.
But, when I am echoing the line, it is... (2 Replies)
Discussion started by: chiru_h
2 Replies
3. UNIX for Dummies Questions & Answers
i need help..!!!!
i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows.
on OS SUN Solaris version 8
How i can remove first line of the text file.
Please suggest me for solutions.
Thank you very much in advance:) (5 Replies)
Discussion started by: madoatz
5 Replies
4. Shell Programming and Scripting
Hi,
How can I edit a line in a file?
For example, a.txt contains:
start: 1 2 3 4
stop: a b c d
and I want to change "3" to "9"
and to add "5" after "4"
the result should be (a.txt):
start: 1 9 3 4 5
stop: a b c d
Thanks,
zed (5 Replies)
Discussion started by: zed
5 Replies
5. Shell Programming and Scripting
I have file which contains around 5000 lines.
The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters.
I want to delete the lines
a> if it starts with 1 and if 576th postion is a digit i,e 0-9
or
b> if it starts with 0 or 9(i,e header and footer)
... (4 Replies)
Discussion started by: millan
4 Replies
6. Shell Programming and Scripting
Hello All,
My file content is:
DROP TABLE
"FACT_WORLD";
CREATE TABLE "FACT_WORLD" (
"AR_ID" INTEGER NOT NULL,
"ORG_ID" INTEGER NOT NULL
)
DATA CAPTURE NONE
COMPRESS YES;
I want to change this file to have entries in one... (6 Replies)
Discussion started by: akash2508
6 Replies
7. Shell Programming and Scripting
Hi Experts,
I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies
8. Shell Programming and Scripting
Hi friends,
Issue1:
I have a text file with the first line like this
#chrom start end Readcount_A Normalized_Readcount_A ReadcountB Normalized_Readcount_B Fc_A_vs_B pvalue_A_vs_B FDR_A_vs_B Fc_B_vs_A pvalue_B_vs_A FDR_B_vs_A <a href="http://unix.com/">Link</a>
How can I change it to the... (11 Replies)
Discussion started by: jacobs.smith
11 Replies
9. UNIX for Dummies Questions & Answers
Hi ,
I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options
Regards
Raj (1 Reply)
Discussion started by: rajeevm
1 Replies
10. Shell Programming and Scripting
Hi.
Pardon me if I'm posting a duplicate thread but..
I have a text file with over 150 Million records, file size is in the range if MB(close to GB).
The requirement is to read ALL the lines excepting the FIRST LINE which is the file header and the LAST LINE which is it's trailer record.
... (8 Replies)
Discussion started by: kumarjt
8 Replies
CDB(5) BSD File Formats Manual CDB(5)
NAME
cdb -- format of the constant database
DESCRIPTION
The cdb database format provides a space-efficient (key,value) database. The format doesn't allow updates in any convenient form. The file
overhead is around 5 bytes per key and 5 bytes per entry. Keys are not stored and it is the responsibility of the caller to validate
matches. The index structure is based on a minimal perfect hash table, so exactly one entry has to be checked for a match.
General Format
The header record of a cdb database consists of the following:
struct header_cdb {
uint8_t magic[7];
uint8_t version;
uint8_t description[16];
uint32_t data_size;
uint32_t entries;
uint32_t entries_index;
uint32_t seed;
};
All fields are in Little Endian byte order.
This is followed by a description of the hash function of entries_index records. The size of each index entry is the logarithm of entries to
base 256, rounded up.
The index records are followed by the start offsets of the entries, followed by data_size. The offsets are relative to the end of the offset
record table and are monotically increasing. The size of each offset record is the logarithm of data_size to base 256, rounded up.
The offset table is followed by the entries in order. No separation or padding is added.
Limitations
The cdb file format is by design intended for a database that can be mapped into memory. The hard limit for the number of entries and keys
is 3435973836. The total size of all values must be smaller than 4GiB.
SEE ALSO
cdbr(3), cdbw(3)
HISTORY
Support for the cdb format first appeared in NetBSD 6.0.
AUTHORS
The cdbr and cdbw functions have been written by Joerg Sonnenberger <joerg@NetBSD.org>.
BSD
April 27, 2010 BSD