C++ getline, parse and take first tokens by condition


 
Thread Tools Search this Thread
Top Forums Programming C++ getline, parse and take first tokens by condition
# 8  
Old 09-18-2014
Your code certainly does not compile for me. I'm still fixing compiler errors.

If it is crashing you forgot to include error checking. I'll add that too.

Why bother storing all of them? If your FASTA file happens to be 3 gigabytes that will be quite a lot of memory in your map!

Also, map is not necessarily array, you're not guaranteed to get the same order out as you put in.

In the end, I think your first solution was better -- just print what you need to print, when you need to print it, and don't keep anything else.

C version:

Code:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define CHUNK 4096
#define TOKEN " \r\n\t" // Includes newlines and \r since fgets includes them

int main(int argc, char *argv[])
{
        int bufsize=CHUNK;
        // Flag variable.  First loop it holds nothing, all other loops,
        // holds "\n" to print a newline before >...
        char *prefix="";
        // Start with a CHUNK-sized buffer that can be enlarged at need
        char *buf=malloc(bufsize);
        FILE *fp;

        if(argc != 2) {
                fprintf(stderr, "No file given\n");
                exit(1);
        }

        fp=fopen(argv[1], "r");
        if(fp == NULL)
        {
                fprintf(stderr, "Can't open %s\n", argv[1]);
                return(1);
        }

        while(fgets(buf, bufsize, fp) != NULL)
        {
                char *tok;
                size_t len=strlen(buf);
                // Increase buffer size if fgets didn't get a complete line
                // complete, as in, ends in '\n'
                while(  buf[len-1] != '\n')
                {
                        buf=realloc(buf, bufsize+=CHUNK); // Make buffer bigger
                        // Read the rest of the line
                        if(fgets(buf+len, CHUNK, fp) == NULL) break;
                        // Count the new length
                        len+=strlen(buf+len);
                }

                tok=strtok(buf, TOKEN);
                if(tok == NULL) // Something strange happened -- no line?
                        continue;

                while(tok != NULL)
                {
                        if(tok[0] == '>')
                        {
                                printf("%s%s ", prefix, strtok(buf, TOKEN));
                                prefix="\n";
                                break; // Leave the loop to ignore all other tokens on this line
                        }
                        fputs(tok,stdout);
                        tok=strtok(NULL, TOKEN);
                }
        }

        fputc('\n', stdout);
        fclose(fp);
}


Last edited by Corona688; 09-18-2014 at 03:23 PM..
This User Gave Thanks to Corona688 For This Post:
# 9  
Old 09-18-2014
Here is your corrected code Smilie

Code:
#include <iostream>
#include <fstream>
#include <string>
#include <map>

/**
 * You need string.h for strtok and strcpy.  MANDATORY!
 * Not having the right headers can cause a CRASH!
 */
#include <string.h>

using namespace std;

int main()
{
   ifstream inFILE("infile.fasta");
   /* You're not using this */
   //int inGuard = 1;               //using a guard variable

   /**
    * If you put it inside the loop, it goes out of scope every loop.
    * That's good when you want that, and bad when you don't.
    * Since you want the value to stay the same every loop, you don't.
    */
   string entryID;

   map <string, string>FastaSeq;   //Declare a map to hold each sequence entry

   while (inFILE.good()) {
      string line;        //declare string for each line
      /* moved above */
      //string entryID, sequence;    //declare two strings for key and value for map
      char *sPtr;        //Declare char pointer sPtr for tokens
      getline(inFILE, line);    //Read the whole line

      //Initialize char pointer sArray for conversion of the string to char*    
      char *sArray = new char[line.length() + 1];
      strcpy(sArray, line.c_str());

      if (sArray[0] == '>') {
         sPtr = strtok(sArray, " ");    //Using space as delimiter get the first token.
         /**
          * If your program crashes, odds are you won't see anything printed to cout.
          * use cerr for debugging instead, it prints instantly instead of being held for later.
          *
          * Use cerr for errors/debugging, cout for data output.
          */
         //cout << sPtr << " ";       //Print the first token only
         cerr << endl << sPtr << " ";
         entryID = sPtr;             //assign the first token as key for the map
         continue;
     } else  {
         sPtr = strtok(sArray, " ");    //Get all the tokens with " " as delimiter.

         /**
          * Always, always, always check your pointers!
          * Never assume strtok must have worked.
          * This is what broke your last 3 programs.
          */
         //FastaSeq[entryID] += sPtr;   // assign first part of sequence to map

         /**
          * The loop checks for NULL, so inside, sPtr is safe to use.
          */
         while (sPtr != NULL) {          //For all tokens
//            cout << sPtr;
            cerr << sPtr;
            FastaSeq[entryID] += sPtr;   // assign more token to sequence
            sPtr = strtok(NULL, " ");
         }
      }

      delete [] sArray;      /* NOT OPTIONAL! */
   }

   cerr << endl << endl;
   inFILE.close();

   //print the map
   map <string, string>::const_iterator seq_itr;

   /**
    * You made an iterator but didn't point it to anything.
    * This is bad for the same reason an unchecked pointer to
    * nothing is bad.
    *
    * Imagine a loop like for(x=0; x != 10; x++) but it's not an int,
    * instead you use z.begin() and z.end().  ++ still works.
    */
   seq_itr=FastaSeq.begin();

//   if (seq_itr != FastaSeq.end()){
   while(seq_itr != FastaSeq.end()) {
      cout << seq_itr->first << " ";

      /**
       * ???  Not sure what you're trying to do here.
       * You can't print an iterator, just its contents (first, second)
       */
      // cout << seq_itr << seq_itr->second << endl;
      cout << seq_itr->second << endl;

      /**
       * You can call ++ on an iterator, it's effectively i=i.next();
       */
      seq_itr++;
   }

    return 0;
}


Last edited by Corona688; 09-18-2014 at 04:14 PM..
This User Gave Thanks to Corona688 For This Post:
# 10  
Old 09-18-2014
The reason to save them into map is for later retrieval.
Say, there are millions of entries (that's exactly any small lab would have!), but only ten or hundreds need be retrieved from it.
After some reading, it seems current programs take two steps:
1) index the dataset;
2) retrieve subsample from the indexed dataset.
It seems to me a hash_map was used. This morning I was reviewing the codes we discussed and thought a program could do the job this way:
Code:
./prog dataset.file sample.list

where sample.list only have the sequence names, i.e. the keys of the map.
sample.list:
Code:
seq01
seq03
seq99 (not in the dataset)

Does this make any sense to you? Or what I missed?
I tried this for the tab-delimited format file, which worked fine but that is not general. If it is a tab-delimited file, the job can be done with the awk script, even grep can do the job easily. However, it seems not easy with grep for the generic format. Thanks.
---------------
You are so fast! While I was writing your second one popped out. Thanks a lot!

---------- Post updated at 03:30 PM ---------- Previous update was at 03:22 PM ----------

Code:
while(seq_itr != FastaSeq.end())
 { cout << seq_itr->first << " ";        
   cout << seq_itr->second << endl;        /* You can call ++ on an iterator, it's effectively i=i.next();        */       
seq_itr++;    
}

I was trying to print each key and value of the map, i.e. the pair of seqID vs. sequence.
# 11  
Old 09-18-2014
Quote:
Originally Posted by yifangt
The reason to save them into map is for later retrieval.
Say, there are millions of entries (that's exactly any small lab would have!), but only ten or hundreds need be retrieved from it.
I see, I see. Hmmm.

How about, instead of storing the entire file, store the locations you've found things. That's your "index". Then, when asked for that information, seek to that spot in the file and read it.

A map is probably not the best data structure for this. A map is probably array or list-based, so if you have 2-million sequences, map["mysequence"] takes a 2-million item loop to tell whether it has it. A tree or a hash would be good. I never got the hang of trees in C++, though, and C++ doesn't have a generic hash table type (unless they added one while I wasn't looking).

On the other hand -- if you know what items you want, why not just print them?
This User Gave Thanks to Corona688 For This Post:
# 12  
Old 09-18-2014
Understand that:
How about, instead of storing the entire file, store the locations you've found things. That's your "index". Then, when asked for that information, seek to that spot in the file and read it.


Isn't that the same to loop/hash the map? And is it do-able?

On the other hand -- if you know what items you want, why not just print them?

Two things there:
1) I do not know if the entry is in the dataset or not,
2) If it is there, I want to get full information (sequences may be stored in unknown number of rows!) of that entry, so that need use a program.

I am aware bioperl/biopython is better to do this type of job, but I am catching C++. And C++ is way faster than perl for sure for millions of queries.

Last edited by yifangt; 09-18-2014 at 04:48 PM..
# 13  
Old 09-18-2014
Quote:
Originally Posted by yifangt
Isn't that the same to loop/hash the map?
Knowing where in 10 gigs of data your information is, and keeping all that 10 gigs of data in memory whether you need it or not, are somewhat different.

Quote:
On the other hand -- if you know what items you want, why not just print them?

Two things there:
1) I do not know if the entry is in the dataset or not
2) If it is there, I want to get full information (sequences may be stored in unknown number of rows!) of that entry, so that need use a program.
OK, now I see the situation.

But I still think you have it backwards. Whenever an idea begins with "store the universe in memory, then use a tiny part of it" my hackles go up. Keep a list of the things you want to find. Scan the file and print only those without storing the universe.

Quote:
I am aware bioperl/biopython is better to do this type of job, but I am catching C++. And C++ is way faster than perl for sure for millions of queries.
I think I mentioned, long ago, a thread on this forum where the OP was using C++ for text processing. But he kept wanting to do more and more with it -- to the point it had rudimentary expressions. In the end it was still a little faster than awk, but it wasn't that fast.

awk, perl, and python are all written in C or C++. If they're slower than your programs, it's because your program does a whole lot less.

awk honestly sounds great for the job here. If your awk program is short, awk will run fast. It already has a very fast array that's based on a hash or tree.
This User Gave Thanks to Corona688 For This Post:
# 14  
Old 09-18-2014
You seems to know every tiny corner of my mind! I am not fluent using any of those programming languages, so that my comments on speed does not count at all.
My colleague simply said to me:"You are overthinking it!" or "You are resistant to this approach!" whenever I ask technique details for things like this.
For this practice, I am struggling to catch the flow of the
Code:
 sPtr = strtok(NULL, " ")

Regular books seldom address this part in great detail. When I took the CS200 course, the professor always emphasized "C and only C, no OOP allowed!"
Now I realized what he meant, seriously!
The part I am still not sure is:
1) In the line with ">", the first field is stored as one string, except the '>' char which is a separator for each record (like RS in awk).
2) All the rest of the field next to the ">" line are concatenated to have a single string. It is easy for printing, but to track them in memory with
Code:
 sPtr = strtok(NULL, " ")

I am not sure at all. Neither am I with this line:
Code:
FastaSeq[entryID] += sPtr;   // assign more token to sequence

For example, the entry:
Code:
>seq01 some description protein 
AGCTAC GTACAT C
AGTCGTGT GAT 
CGAGC GGG

Only seq01 is picked up for key on the first line, the other part are discarded; from the second row of the entry all is concatenated: AGCTACGTACATCAGTCGTGTGATCGAGCGGG for value of the map (if I insist map be used!)

I seem to understand the syntax, as I can print out the individual field parsed, but do not know how to combine certain fields together if needed. Maybe I should not say I understand the syntax.
How the pointer/reference is manipulated behind is the bottleneck for me to catch the whole point. Can you elaborate that? Thanks!

Last edited by yifangt; 09-18-2014 at 06:10 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parse xml in shell script and extract records with specific condition

Hi I have xml file with multiple records and would like to extract records from xml with specific condition if specific tag is present extract entire row otherwise skip . <logentry revision="21510"> <author>mantest</author> <date>2015-02-27</date> <QC_ID>334566</QC_ID>... (12 Replies)
Discussion started by: madankumar.t@hp
12 Replies

2. Programming

Reading tokens

I have a String class with a function that reads tokens using a delimiter. For example String sss = "6:8:12:16"; nfb = sss.nfields_b (':'); String tkb1 = sss.get_token_b (':'); String tkb2 = sss.get_token_b (':'); String tkb3 = sss.get_token_b (':'); String tkb4 =... (1 Reply)
Discussion started by: kristinu
1 Replies

3. Shell Programming and Scripting

Parse tab delimited file, check condition and delete row

I am fairly new to programming and trying to resolve this problem. I have the file like this. CHROM POS REF ALT 10_sample.bam 11_sample.bam 12_sample.bam 13_sample.bam 14_sample.bam 15_sample.bam 16_sample.bam tg93 77 T C T T T T T tg93 79 ... (4 Replies)
Discussion started by: empyrean
4 Replies

4. Shell Programming and Scripting

Need tokens in shell script

Hi All, Im writing a shell script in which I want to get the folder names in one folder to be used in for loop. I have used: packsName=$(cd ~/packs/Acquisitions; ls -l| awk '{print $9}') echo $packsName o/p: opt temp user1 user2 ie. Im getting the output as a string. But I want... (3 Replies)
Discussion started by: AB10
3 Replies

5. Shell Programming and Scripting

+: more tokens expected

Hey everyone, i needed some help with this one. We move into a new file system (which should be the same as the previous one, other than the name directory has changed) and the script worked fine in the old file system and not the new. I'm trying to add the results from one with another but i'm... (4 Replies)
Discussion started by: senormarquez
4 Replies

6. Shell Programming and Scripting

Replacing tokens

Hi all, I have a variable with value DateFileFormat=NAME.CODE.CON.01.#.S001.V1.D$.hent.txt I want this variable to get replaced with : var2 is a variable with string value DateFileFormat=NAME\\.CODE\\.CON\\.01\\.var2\\.S001\\.V1\\.D+\\.hent\\.txt\\.xml$ Please Help (3 Replies)
Discussion started by: abhinav192
3 Replies

7. Shell Programming and Scripting

Shell script to parse/split input string and display the tokens

Hi, How do I parse/split lines (strings) read from a file and display the individual tokens in a shell script? Given that the length of individual lines is not constant and number of tokens in each line is also not constant. The input file could be as below: ... (3 Replies)
Discussion started by: yajaykumar
3 Replies

8. Shell Programming and Scripting

: + : more tokens expected

Hello- Trying to add two numbers in a ksh shell scripts and i get this error every time I execute stat1_ex.ksh: + : more tokens expected stat1=`cat .stat1a.tmp | cut -f2 -d" "` stat2=`cat .stat2a.tmp | cut -f2 -d" "` j=$(($stat1 + $stat2)) # < Here a the like the errors out echo $j... (3 Replies)
Discussion started by: Nomaad
3 Replies

9. UNIX for Advanced & Expert Users

How to parse through a file and based on condition form another output file

I have one file say CM.txt which contains values like below.Its just a flat file 1000,A,X 1001,B,Y 1002,B,Z ... .. total around 4 million lines of entries will be in that file. Now i need to write another file CM1.txt which should have 1000,1 1001,2 1002,3 .... ... .. Here i... (6 Replies)
Discussion started by: sivasu.india
6 Replies

10. UNIX for Dummies Questions & Answers

tokens in unix ?

im trying to remove all occurences of " OF xyz " in a file where xyz could be any word assuming xyz is the last word on the line but I won't always be. at the moment I have sed 's/OF.*//' but I want a nicer solution which could be in pseudo code sed 's/OF.* (next token)//' Is... (6 Replies)
Discussion started by: seaten
6 Replies
Login or Register to Ask a Question