Visit Our UNIX and Linux User Community


Is there a system call other than 'open' for opening very large files?


 
Thread Tools Search this Thread
Top Forums Programming Is there a system call other than 'open' for opening very large files?
# 1  
Old 08-15-2009
Question Is there a system call other than 'open' for opening very large files?

Dear all,


Inside a C program, I want to open a very big file (about 12 GB) in order to read its
content. Here is the code:

Code:
    /*
          argv[1] contains the path to the file.
    */
    inputFileDescriptor = open(argv[1], O_RDONLY);
    if (inputFileDescriptor < 0)
    {
        fprintf(stderr, "Error: Could not open the input file\n");
        fprintf(stderr, "%s\n", strerror(errno));
        return 1;
    }

And it seems that the file is too big, because I receive the following error message:
Code:
Error: Could not open the input file
Value too large for defined data type

Is there any other system call, allowing to open very big files?


Thanks in advance,
Dariyoosh
Smilie
# 2  
Old 08-15-2009
Your system should have something like fopen64() or something that enables largefiles.

try man largefile (or man largefiles), which will show you what support is available. In some cases setting an environment variable or a cc option (both during compile time) allows open to access huge files.

You did not mention your OS. That would help - a lot.
# 3  
Old 08-15-2009
Try this:

Code:
open( filename, O_RDONLY | O_LARGEFILE );

You also probably need to add these macro definitions to your compiles:

Code:
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64

Or you can just compile everything as a 64-bit binary if you're running on a 64-bit OS. If you go that route it's usually a lot simpler. But you have to actually read specifications, because among other common problematic coding practices I've seen far too many times, pointers aren't integers when you compile 64-bit apps.

(Well, pointers are never, ever ints, but because most 32-bit memory models happen to make both pointers and ints 32 bits, it works most of the time. For 32-bit apps, anyway. An "int" is an "int", a pointer is a pointer, and a size_t is a size_t.)
# 4  
Old 08-16-2009
Dear jim mcnamara and achenle


Thank you very much both of you for your help.
I tried with GCC command line option -D_FILE_OFFSET_BITS=64
and it worked.

currently I'm using ubuntu 9.0.4 32-bit
but I think I'm going to switch to Fedora Core 11 (64-bit)
which according to what I read in forums it is better
configured for application development (please correct
me if I'm wrong)

Just one more question about what achenle mentioned.
Quote:
... Or you can just compile everything as a 64-bit binary
if you're running on a 64-bit OS. If you go that route
it's usually a lot simpler. But you have to actually read specifications ...
Is there any specification written particularly for 64 bit?
I mean, If I install a 64bit Linux, will I have linux man pages
specially for 64bit or there will be the same 32bit linux manual
pages when I type man <section> <command>?

Because currently the only official specification that I use
which is also available on the web is the last version of POSIX:

The Open Group Base Specifications Issue 7

It seems that in the case where everything fully conforms to this
specification, the application will be portable (which seems that it is
not the case in my problem, because neither 'fopen64' nor
-D_FILE_OFFSET_BITS=64 is POSIX compliant).


Kind Regards,
Dariyoosh
Smilie
# 5  
Old 08-16-2009
Quote:
Originally Posted by dariyoosh
Just one more question about what achenle mentioned.
Is there any specification written particularly for 64 bit? I mean, If I install a 64bit Linux, will I have linux man pages specially for 64bit or there will be the same 32bit linux manual pages when I type man <section> <command>?
Are you writing your replies in notepad then copy-pasting or something? That's not necessary, and it introduces millions of surplus linebreaks.

64-bit wouldn't need a special call for opening very long files, because a few important fundamental types would already be 64-bit in the first place hence would never hit the 32-bit integer limit. The specification is the same since the specification doesn't enforce type sizes, that's a hardware limit that 32-bit UNIX ran into and had to work around in a vaguely ugly way.
# 6  
Old 08-16-2009
The only thing I'd add at this point is the if you start developing a 64-bit application is to be real careful with your pointers, offsets, and sizes. If you've been programming for any time in strictly 32-bit apps, you've probably developed habits that cause you to at least implicitly treat those as 32-bit values, if you don't do it explicitly already.

I'm not sure offhand what GCC options would help ("-Wall", maybe?), but I doknow later versions of Sun's Studio C++ compiler support options such as "-xport64" and "+w2" that will cause the compiler to emit warnings for things like 64-to-32-bit truncations and code that depends on sign extension rules. Those warning options were put in to help porting code to 64-bit environments.

FWIW, I always like all my code to be completely clean of even the slightest warning - at the highest warning level whatever compiler I'm using supports. I figure that if the people who wrote the compiler that I'm using to convert my code into a binary think what I'm trying to do is a bad enough idea to take the time to warn me about it, it's probably something I shouldn't be doing.
# 7  
Old 08-17-2009
Many thanks for the answers!

Kind Regards,
Dariyoosh
Smilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #559
Difficulty: Easy
A runtime error is an error that occurs while a program is being executed.
True or False?

10 More Discussions You Might Find Interesting

1. Solaris

Getting read only propmt when opening /etc/system file

root@atrcx146:/# vi /etc/system "/var/tmp/Exv9a4Rb" Read-only file system Please let me know the reason (1 Reply)
Discussion started by: Marty11
1 Replies

2. Shell Programming and Scripting

How to open large datafie in awk?

I just tried awk '{print}' all.plo awk: cannot open all.plo (Value too large for defined data type)awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}' all.plo awk: cannot open all.plo (Value too large for defined data type) datafile size is 4.8GB any other provision ? only cat works FS is... (4 Replies)
Discussion started by: Akshay Hegde
4 Replies

3. Programming

open() system call in c++????

Hi friends, I am trying to use the open system call in c++ language. Please have a look at my code. vi Open.cpp 1 #include <stdio.h> 2 #include <iostream> 3 4 #define BUFSIZE 1 5 6 using namespace std; 7 8 int main() 9 { 10 ... (5 Replies)
Discussion started by: gabam
5 Replies

4. UNIX for Advanced & Expert Users

resize2fs open: Is a directory while opening /home

Can anyone explain this error to me? I am trying to use resize2fs to resize an ext4 partition. I did unmount the partition and I don't have any files opened or anything like that. Here is the full output. # resize2fs -p /home 3G resize2fs 1.41.12 (17-May-2010) open: Is a directory while... (1 Reply)
Discussion started by: cokedude
1 Replies

5. SCO

Need advice: Copying large CSV report files off SCO system

I have a SCO Unix server from 1999 running SCO 5.0.5 and some ancient accounting software called Real World A report writer program on the system is used to generate CSV files from accounting that we write with DOSCOPY commands to 3.5" floppies In the next 60 days we will be decommissioning... (11 Replies)
Discussion started by: magnetman
11 Replies

6. Shell Programming and Scripting

Assistance pls - pipe error: Too many open files in system

When I run a bash script in the customer system, it throws the warning and script exits Exec '/root/sample.sh' @ hostname-- OK (warn) /root/sample.sh: pipe error: Too many open files in system /root/sample.sh: n + : syntax error: operand expected (error token is " ") Exec... (5 Replies)
Discussion started by: vidhyamirra
5 Replies

7. UNIX for Advanced & Expert Users

System Call Wrapper of 'open'

When the programmer use 'open' function, the process is like below. "open -> system call wrapper of open in Glibc -> syscall_open in kernel" I found the wrapper of open, but there is no implementation like 'int $80'. int __open (file, oflag) const char *file; int oflag; { ... (3 Replies)
Discussion started by: yuno96
3 Replies

8. Web Development

Content Management System for uploading large files

Hi everybody, I am currently trying to develop a simple content management system where I have an internal website for my users to upload large files onto the server. The site is password protected and my users won't be trying to hack into the system so security is a non-factor (as least for... (3 Replies)
Discussion started by: z1dane
3 Replies

9. UNIX for Advanced & Expert Users

how to differentiate system call from library call

Hi, Ho do I differentiate system call from library call? for example if I am using chmod , how do I find out if it is a system call or library call? Thanks Muru (2 Replies)
Discussion started by: muru
2 Replies

10. AIX

Why is my file system cache so large

Hi I have a filesystem cache which is around 20G in size and I'm a bit perplexed as to what is in it. I'm running Sybase on the machine with the db on raw volumes and a tempdb on a ramdisk. My understanding is that raw volumes are not cached and I assumed that the ramdisk is not either. Am... (1 Reply)
Discussion started by: mgibbons
1 Replies

Featured Tech Videos