Sponsored Content
Top Forums Programming Fast string removal from large text collection Post 302561092 by achenle on Monday 3rd of October 2011 06:28:54 AM
Old 10-03-2011
1. Stop using malloc()/calloc() and free() every time you need memory. Get ONE chunk of memory and reuse it. For example, pass a character buffer into a method instead of using calloc() to allocate a new one each and every time.
2. Fix your memory leaks - I spotted at least two, on in file_name_generator(), one caused by the return value of file_name_generator() overwriting a malloc()'d pointer.
3. Don't EVER use fgetc().
4. Don't read files TWICE. Use something like fgets() and process each word as you read it. If you're using rewind(), you've done something wrong.
This User Gave Thanks to achenle For This Post:
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Large Text Files

Hi All I have approximately 10 files that are at least 100+ MB in size. I am importing them into a DB to output them to the web. What i need to do first is clean the files up so i dont have un necessary rows in the DB. Below is what the file looks like: Ignore the <TAB> annotations as that... (4 Replies)
Discussion started by: caddyjoe77
4 Replies

2. Programming

fopen() + reading in large text files

For reading in large text files (say files over 1kB in size) are there any issues with fopen() that I should be aware of ? cheers (2 Replies)
Discussion started by: JamesGoh
2 Replies

3. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

4. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

5. UNIX for Dummies Questions & Answers

selective removal of blank spaces in string

Hi, I'm a newbie to shell scripting and I have the following problem: I need all spaces between two letters or a letter and a number exchanged for an underscore, but all spaces between a letter and other characters need to remain. Searching forums didn't help... One example for clarity: ... (3 Replies)
Discussion started by: Cpt_Cell
3 Replies

6. Shell Programming and Scripting

Large XML to MySQL - fast way

Hello, Sorry for my bad english. I need to improve performance in project managing large data, these data are exported to a MySql from XML. Now I use PHP (XMLReader ()) to do this job. I need a faster way to do this process. Which do you think is the best way? Example: (the item... (2 Replies)
Discussion started by: stendelis
2 Replies

7. Shell Programming and Scripting

String removal from file

Dear all From below mention input file I needed op file as show below. I am using below code but not worked. I/p file BSCBCH1 EXAL-1-4 WO* SMPS MAINS FAIL BSCBCH1 EXAL-1-5 WO* SMPS RECTIFIER FAIL BSCBCH1 EXAL-1-6 WO* SMPS MAJOR ALARM BSCBCH2 EXAL-1-10 WO* ... (5 Replies)
Discussion started by: jaydeep_sadaria
5 Replies

8. UNIX for Dummies Questions & Answers

awk string removal

Hi, I am trying to remove a string ".var" using the below command but it's not working as expected, when I execute this in the command prompt using the echo it's working fine , please let me know where I am doing it wrong. UYRD=$FILE_$timestamp.csv | awk '{gsub(".var", "");print}' # this is... (6 Replies)
Discussion started by: shruthidwh
6 Replies

9. Shell Programming and Scripting

Removing string from CSV file by provide removal string from other file

What I need is to remove the text from Location_file.txt from each line matching all entries from Remove_location.txt Location_file.txt FlowPrePaid, h3nmg1cm2,Jamaica_MTAImageFileFlowPrePaid,h0nmg1cm1, Flow_BeatTest,FlowRockTest FlowNewTest,FlowNewTest,h0nmg1cm1 PartiallySubscribed,... (3 Replies)
Discussion started by: ketanraut
3 Replies
MALLOC(3)						     Library Functions Manual							 MALLOC(3)

NAME
malloc, free, realloc, calloc, alloca - memory allocator SYNOPSIS
char *malloc(size) unsigned size; free(ptr) char *ptr; char *realloc(ptr, size) char *ptr; unsigned size; char *calloc(nelem, elsize) unsigned nelem, elsize; char *alloca(size) int size; DESCRIPTION
Malloc and free provide a general-purpose memory allocation package. Malloc returns a pointer to a block of at least size bytes beginning on a word boundary. The argument to free is a pointer to a block previously allocated by malloc; this space is made available for further allocation, but its contents are left undisturbed. Needless to say, grave disorder will result if the space assigned by malloc is overrun or if some random number is handed to free. Malloc maintains multiple lists of free blocks according to size, allocating space from the appropriate list. It calls sbrk (see brk(2)) to get more memory from the system when there is no suitable space already free. Realloc changes the size of the block pointed to by ptr to size bytes and returns a pointer to the (possibly moved) block. The contents will be unchanged up to the lesser of the new and old sizes. In order to be compatible with older versions, realloc also works if ptr points to a block freed since the last call of malloc, realloc or calloc; sequences of free, malloc and realloc were previously used to attempt storage compaction. This procedure is no longer recommended. Calloc allocates space for an array of nelem elements of size elsize. The space is initialized to zeros. Alloca allocates size bytes of space in the stack frame of the caller. This temporary space is automatically freed on return. Each of the allocation routines returns a pointer to space suitably aligned (after possible pointer coercion) for storage of any type of object. If the space is of pagesize or larger, the memory returned will be page-aligned. SEE ALSO
brk(2), pagesize(2) DIAGNOSTICS
Malloc, realloc and calloc return a null pointer (0) if there is no available memory or if the arena has been detectably corrupted by stor- ing outside the bounds of a block. Malloc may be recompiled to check the arena very stringently on every transaction; those sites with a source code license may check the source code to see how this can be done. BUGS
When realloc returns 0, the block pointed to by ptr may be destroyed. The current implementation of malloc does not always fail gracefully when system memory limits are approached. It may fail to allocate memory when larger free blocks could be broken up, or when limits are exceeded because the size is rounded up. It is optimized for sizes that are powers of two. Alloca is machine dependent; its use is discouraged. 4th Berkeley Distribution May 14, 1986 MALLOC(3)
All times are GMT -4. The time now is 02:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy