Programming

Hello,

I am not that experienced with Linux, and I am currently facing some issues.

The application I'm working on uses hundreds of threads. To optimize the memory usage, I am putting all my data inside a shared object (so).

The steps for this are as follows:

1. a C file (generated from an XML file) with lots of arrays (const char c1[] ="string" and const char* arr_const[] = {"s1", "s2", ...."sN"}, etc.)

2. a shell script to compile this to a shared object:
gcc -g -m64 -Wall -fPIC -c bigfile.c
gcc -m64 -shared -Wl -o <custom_new_name>.so bigfile.o

3. some code (on each thread) to open the shared object and get the refferences to my arrays from the shared object:
void ** handle;
*handle=dlopen (<full_path_to_so>, RTLD_NOW);
char** local_var = (char**) dlsym(*handle, "arr_const");

NOTE: the handles don't get unloaded until my application closes.

Using shared objects is mandatory for two reasons:
- optimized memory usage when launchig thousands of threads
- the ability to create new shared objects from new XML files and adding them to the application after it has been installed on client machines (with the help of text config files).

My problem is that the process described above does not work on my x86_64 CentOS 5.8 (dlopen returns NULL). The entire application is built with 64bit settings.
When using this on CentOS 32bit, with gcc using -m32 instead of -m64 for compiling the shared objects, and building my app with 32bit settings, everything is fine.

Any ideas ?
Thank you

I usually see gcc's -fPIC option used in Linux x86_64 architecture, the 64-bit loader may expect relocatable code. Try it.

I already used -fPIC when making the .o object.
I'll redeploy my app on 64bit architecture and add the -fPIC option to the second call to gcc, and let you know how that went.

Try perror after dlopen fails, so you can see why it fails.

There's another way you can share huge gobs of text among programs without wasting memory, by the way -- map it in.

Code:

struct stat buf;
int fd=open("/path/to/file", O_RDONLY);
void *mem;

fstat(fd, &buf);

// Read-only file mapping with shared memory
mem=mmap(NULL, buf.st_size, PROT_READ, MAP_SHARED, fd, 0);
if(mem != MAP_FAILED)
{
        printf("Mapped %d bytes of memory at %p\n", buf.st_size, mem);
}

// The contents of the file 'fd' are now available inside the block of memory
// pointed to by 'mem'.

Of course, this doesn't give you handy variables and arrays the way a shared object does, just a big block of raw data, but should be very portable.

Thanks, but we've already looked into this, and it would give us a lot of trouble in case we need debugging, or changing the ridiculous amount of arrays in the shared object.

right now I am looking at 3MB to 6MB of generated C code

EDIT: besides, I wouldn't want to trash my work for the past 3 weeks (including tons of nifty macros and other code changes I had to do because of switching to shared libraries), and go back to the drawing board with my tale between my legs

Try perror after dlopen fails, so you can see why it fails.

perror or dlerror ?
I tried dlerror so far, but nothing interesting there Didn't think about perror yet, I'll give it a try

EDIT:
Is there anything wrong with the way I build my .so file ?
It seems something crashes when trying to open it.