Login or Register to Ask a Question and Join Our Community


Programming

Reading a particular line from a .txt file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Programming Reading a particular line from a .txt file
# 1  
Old 05-30-2010
Data Reading a particular line from a .txt file
Hi,
I have a .txt file which contains the x, y and z co-ordinates of particles which I am trying to cast for a particular compound. The no. of particles present is of the order of 2 billion and hence the size of the text file is of the order of a few Gigabytes. The particles have been casted layer wise - thus, if there are 15000 layers in which I have casted the particles, there are approx. 2 billion/15000 particles in each layer. Thus, every 2 billion/15000 particles have the same Y co-ordinate. Now, I need to read the particles at a given value of Y (say y = 10). I wrote a small program, where I had used fin.seekg( ). However I realized that the seeking of the position from where the file has to be read is not done line-wise, but is done character-wise. Could someone please tell me how I could start seeking from a particular line in the file using a simple C++ program.
# 2  
Old 05-30-2010
When the lines are a fixed record length, you can call seek() or fseek() in C++, C to get to a known line position.
this is c:
Code:
void seek_to_line(FILE *in, const long recl, const long lineno)
{
     fseek(in, lineno * recl, SEEK_SET);
}

This places the file pointer at the beginning of lineno, assuming recl is fixed.

Otherwise you can try to optimize I/O (see Steven's Advanced Programming in the UBIX Environment) by increasing
buffersize
Code:
FILE *in=fopen("somefile, "r");
int lineno=250002;
char tmp[256]={0x0};
char buf[16384]={0x0};

setvbuf(in, buf, 16384, _IOFBF );
while(--lineno)
   fgets(tmp, sizeof(tmp), in);

you have to call setvbuf BEFORE any I/O on the stream
Last edited by jim mcnamara; 05-30-2010 at 12:16 PM..
# 3  
Old 05-30-2010
That's only going to work if you can guarantee each line fits into your buffer.

If you can do that, great. If not, you pretty much have to count newline characters. This works, and should be fairly fast as you'd be relying on the OS to page in the data, which should be fast enough. If it's a really big file, and you know you're only going through it once, it'd be faster to use open() and read() with direct IO set so you bypass any page cache (if you're only looking once at each byte of a bunch of gigabytes of data, any caching is wasted cycles):

Code:
struct stat sb;
int fd = open( filename, O_RDONLY );
fstat( fd, &sb );
char *ptr = mmap( NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0 );
uint64_t offset;
uint64_t line_count = 0;
uint64_t desired_line = 123455;

for ( offset = 0; offset < sb.st_size; offset++ )
{
    if ( '\n' == ptr[ offset ] )
    {
        line_count++;

        if ( line_count == desired_line )
        {
            break;
        }
    }
}

// if offset is less than file size, the desired line was found
if ( offset < sb.st_size )
{
    // line starts at offset + 1
    offset++;
        .
        .
        .
}

Note that has no error checking.
# 4  
Old 05-30-2010
Quote:
Originally Posted by mugga
Hi,
I have a .txt file which contains the x, y and z co-ordinates of particles which I am trying to cast for a particular compound. The no. of particles present is of the order of 2 billion and hence the size of the text file is of the order of a few Gigabytes. The particles have been casted layer wise - thus, if there are 15000 layers in which I have casted the particles, there are approx. 2 billion/15000 particles in each layer. Thus, every 2 billion/15000 particles have the same Y co-ordinate. Now, I need to read the particles at a given value of Y (say y = 10). I wrote a small program, where I had used fin.seekg( ). However I realized that the seeking of the position from where the file has to be read is not done line-wise, but is done character-wise. Could someone please tell me how I could start seeking from a particular line in the file using a simple C++ program.
Considering the layout of your file to be in this form

X Y Z
-5.55 4.44 6.5
10.66 44.5 85.99
.....
......
.....
The values are separated by whitespace(s).

A simple awk liner will rid of of your messy C++ code.

Code:
 awk -v y=  '$2==y{print "x="$1,"z="$3}' file.txt

subsitute the value of y . e.g. awk -v y=4.44 '.........'
output from above - x=-5.55 z=6.5

Quote:
why re-invent wheels?
Hope this helps,
Regards,
GauravImage
# 5  
Old 05-31-2010
Thanks Guys!
Problem solved Smilie
# 6  
Old 05-31-2010
Quote:
Originally Posted by mugga
Thanks Guys!
Problem solved Smilie
How did you solve it. Please share it with us. We would like to know.

Regards,
Gaurav.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

' for reading (No such file or directory) `Solaris_websummary.txt

echo "1.1 Apply latest OS patches;" awk '/1.2 Install/ {P=0} P {print $0} FNR==1{printf("From file %s:\n", FILENAME)} /1.1 Apply/ {P=1}' solarisappsummary.txt solarisdbsummary.txt solaris_websummary.txt echo "1.2 Install TCP Wrappers;" awk '/1.3 Install/ {P=0} P {print $0}... (1 Reply)
Discussion started by: alvinoo
1 Replies

2. UNIX for Dummies Questions & Answers

Split Every Line In Txt Into Separate Txt File, Named Same As The Line

Hi All Is there a way to export every line into new txt file where by the title of each txt output are same as the line ? I have this txt files containing names: Kandra Vanhooser Rhona Menefee Reynaldo Hutt Houston Rafferty Charmaine Lord Albertine Poucher Juana Maes Mitch Lobel... (2 Replies)
Discussion started by: Nexeu
2 Replies

3. Shell Programming and Scripting

Desired output.txt for reading txt file using awk?

Dear all, I have a huge txt file (DATA.txt) with the following content . From this txt file, I want the following output using some shell script. Any help is greatly appreciated. Greetings, emily DATA.txt (snippet of the huge text file) 407202849... (2 Replies)
Discussion started by: emily
2 Replies

4. Shell Programming and Scripting

Need to append the date | abcddate.txt to the first line of my txt file

I want to add/append the info in the following format to my.txt file. 20130702|abcd20130702.txt FN|SN|DOB I tried the below script but it throws me some exceptions. <#!/bin/sh dt = date '+%y%m%d'members; echo $dt+|+members+$dt; /usr/bin/awk -f BEGIN { FS="|"; OFS="|"; } { print... (6 Replies)
Discussion started by: harik1982
6 Replies

5. Shell Programming and Scripting

Reading text file, comparing a value in a line, and placing only part of the line in a variable?

I need some help. I would like to read in a text file. Take a variable such as ROW-D-01, compare it to what's in one line in the text file such as PROD/VM/ROW-D-01 and only input PROD/VM into a variable without the /ROW-D-01. Is this possible? any help is appreciated. (2 Replies)
Discussion started by: xChristopher
2 Replies

6. UNIX for Dummies Questions & Answers

C-Shell script help reading from txt file

I need to write a C-Shell script with these properties: It should accept two arguments on the command line. The first argument is the name of a file which contains a list of names, and the second argument is the name of a directory. For each file in the directory, the script should print the... (1 Reply)
Discussion started by: cerce
1 Replies

7. Shell Programming and Scripting

[Solved] Problem in reading a file line by line till it reaches a white line

So, I want to read line-by-line a text file with unknown number of files.... So: a=1 b=1 while ; do b=`sed -n '$ap' test` a=`expr $a + 1` $here do something with b etc done the problem is that sed does not seem to recognise the $a, even when trying sed -n ' $a p' So, I cannot read... (3 Replies)
Discussion started by: hakermania
3 Replies

8. Shell Programming and Scripting

reading the txt file

hi to all im having some 20,000 files in that im having some contents say the tabulation of biophysics lab readings ... and i want read tat file and look into tat wether a number say -18.90 is there r not .. and if there print tat no wit file name beside thank you:D (1 Reply)
Discussion started by: maximas
1 Replies

9. Shell Programming and Scripting

Reading Characters from txt file

Hello, I am new to shell scripting, and I am trying to create a script that reads an input like the following firstname:lastname:age firstname:lastname:age firstname:lastname:age in a text file. I have a 2 part question. First how do I open the file in a shell script. And then how can... (7 Replies)
Discussion started by: TexasGuy
7 Replies
Login or Register to Ask a Question