Programming

Dear all,


Inside a C program, I want to open a very big file (about 12 GB) in order to read its
content. Here is the code:

Code:

    /*
          argv[1] contains the path to the file.
    */
    inputFileDescriptor = open(argv[1], O_RDONLY);
    if (inputFileDescriptor < 0)
    {
        fprintf(stderr, "Error: Could not open the input file\n");
        fprintf(stderr, "%s\n", strerror(errno));
        return 1;
    }

And it seems that the file is too big, because I receive the following error message:

Code:

Error: Could not open the input file
Value too large for defined data type

Is there any other system call, allowing to open very big files?


Thanks in advance,
Dariyoosh

Your system should have something like fopen64() or something that enables largefiles.

try man largefile (or man largefiles), which will show you what support is available. In some cases setting an environment variable or a cc option (both during compile time) allows open to access huge files.

You did not mention your OS. That would help - a lot.

Try this:

Code:

open( filename, O_RDONLY | O_LARGEFILE );

You also probably need to add these macro definitions to your compiles:

Code:

-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

Or you can just compile everything as a 64-bit binary if you're running on a 64-bit OS. If you go that route it's usually a lot simpler. But you have to actually read specifications, because among other common problematic coding practices I've seen far too many times, pointers aren't integers when you compile 64-bit apps.

(Well, pointers are never, ever ints, but because most 32-bit memory models happen to make both pointers and ints 32 bits, it works most of the time. For 32-bit apps, anyway. An "int" is an "int", a pointer is a pointer, and a size_t is a size_t.)

Dear jim mcnamara and achenle


Thank you very much both of you for your help.
I tried with GCC command line option -D_FILE_OFFSET_BITS=64
and it worked.

currently I'm using ubuntu 9.0.4 32-bit
but I think I'm going to switch to Fedora Core 11 (64-bit)
which according to what I read in forums it is better
configured for application development (please correct
me if I'm wrong)

Just one more question about what achenle mentioned.
Is there any specification written particularly for 64 bit?
I mean, If I install a 64bit Linux, will I have linux man pages
specially for 64bit or there will be the same 32bit linux manual
pages when I type man <section> <command>?

Because currently the only official specification that I use
which is also available on the web is the last version of POSIX:

The Open Group Base Specifications Issue 7

It seems that in the case where everything fully conforms to this
specification, the application will be portable (which seems that it is
not the case in my problem, because neither 'fopen64' nor
-D_FILE_OFFSET_BITS=64 is POSIX compliant).


Kind Regards,
Dariyoosh

Are you writing your replies in notepad then copy-pasting or something? That's not necessary, and it introduces millions of surplus linebreaks.

64-bit wouldn't need a special call for opening very long files, because a few important fundamental types would already be 64-bit in the first place hence would never hit the 32-bit integer limit. The specification is the same since the specification doesn't enforce type sizes, that's a hardware limit that 32-bit UNIX ran into and had to work around in a vaguely ugly way.

The only thing I'd add at this point is the if you start developing a 64-bit application is to be real careful with your pointers, offsets, and sizes. If you've been programming for any time in strictly 32-bit apps, you've probably developed habits that cause you to at least implicitly treat those as 32-bit values, if you don't do it explicitly already.

I'm not sure offhand what GCC options would help ("-Wall", maybe?), but I doknow later versions of Sun's Studio C++ compiler support options such as "-xport64" and "+w2" that will cause the compiler to emit warnings for things like 64-to-32-bit truncations and code that depends on sign extension rules. Those warning options were put in to help porting code to 64-bit environments.

FWIW, I always like all my code to be completely clean of even the slightest warning - at the highest warning level whatever compiler I'm using supports. I figure that if the people who wrote the compiler that I'm using to convert my code into a binary think what I'm trying to do is a bad enough idea to take the time to warn me about it, it's probably something I shouldn't be doing.

Many thanks for the answers!

Kind Regards,
Dariyoosh