Make sure strtok_r() function


 
Thread Tools Search this Thread
Top Forums Programming Make sure strtok_r() function
# 1  
Old 01-16-2014
Make sure strtok_r() function

Hello,
I was trying to understand more on strtok_r() function with following code:
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* *From http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=strtok_r
 A FreeBSD man pages  * */

int main()
{
    char string1[80];
    char *sep = "\\/:;=-";               /*Delimiters*/
    char *word, *tmp1;                   /*For the loop, *word is the return value, tmp1 is the temporary holder of the rest*/

    strcpy(string1, "This;is.a:test:of=the/string\\tokenizer-function.");

    while( (word=strtok_r(string1, sep, &tmp1)) != NULL) {   
        printf("So far we're at %s\n", word);
        string1 = tmp1;                             //Want move string1 to the next token. Error!!!
    }
    return 0;
}

The problem is with the loop at line: string1 = tmp1 where I want move the pointer to the next token
Code:
error: incompatible types when assigning to type ‘char[80]’ from type ‘char *’

Explanation and correction is greatly appreciated.
# 2  
Old 01-16-2014
Make sure to follow proper usage of this function, where on the first call the 1st argument should point to the input string, then on the following calls it should be NULL.
This User Gave Thanks to migurus For This Post:
# 3  
Old 01-16-2014
I have no idea what you're even trying to do with string1 = tmp1;. That operation does not make sense.

Every time you call strtok_r(string1, ...) you are telling it to start over! Remember how strtok works.

Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* *From http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=strtok_r
 A FreeBSD man pages  * */

int main()
{
    char string1[80]="This;is.a:test:of=the/string\\tokenizer-function."; // You can assign directly, without strcpy
    const char *sep = "\\/:;=-";               /*Delimiters*/
    char *word, *tmp1;                   /*For the loop, *word is the return value, tmp1 is the temporary holder of the rest*/

    word=strtok_r(string1, sep, &tmp1);

    while(word != NULL)
    {
        printf("word = '%s'\n", word);
        word=strtok_r(NULL, sep, &tmp1);
    }

    return 0;
}

Code:
$ gcc strtok.c
$ ./a.out

word = 'This'
word = 'is.a'
word = 'test'
word = 'of'
word = 'the'
word = 'string'
word = 'tokenizer'
word = 'function.'

$


Last edited by Corona688; 01-16-2014 at 02:51 PM..
This User Gave Thanks to Corona688 For This Post:
# 4  
Old 01-16-2014
extra pointer

Thanks Corona!
My last thread related to strtok() function in this forum triggered this one , that I had hard time to understand the pointer moving along the line:
Quote:
tok = strtok(NULL, " \r\n\t"); // NULL means 'continue from last token'.
Quote:
strtok_r(3): extract tokens from strings - Linux man page
The strtok() function uses a static buffer while parsing, so it's not thread safe. Use strtok_r() if this matters to you.
The other reason is when I use an extra pointer, it worked fine but I do not know why!
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* Example how to use strtok_r() function
 *
 *From http://www.gsp.com/cgi-bin/man.cgi?section=3&topic=strtok_r
 A FreeBSD man pages
 * */

int main()
{
    char string1[80];
    char *sep = "\\/:;=-";            /*Delimiters*/
    char *word, *tmp1;                /*For the loop, *word is the return value, tmp1 to hold the rest of the parsed line */
    char *ptr;  
 
    //I want strcpy to handle FILE stream line-by-line later, if it is not the wrong way.
    strcpy(string1, "This;is.a:test:of=the/string\\tokenizer-function.");  
    ptr = string1;                    /*Extra variable to hold string1*/
    
    while( (word=strtok_r(ptr, sep, &tmp1)) != NULL) {
        printf("So far we're at %s\n", word);
        ptr = tmp1;  //string1 = tmp1;     //Want move string1 to the next token. Error!!! 
    }
    return 0;
}

So why did it work by using the extra pointer ptr?
# 5  
Old 01-16-2014
It doesn't work "because" of the extra pointer. You found a way around the syntax error, it's valid to assign a pointer to a pointer, but I can't imagine it's actually giving the intended results.

Every time you give strtok_r a string as the first parameter, you are telling it to start over. Keep giving it the same string, and you will keep getting the same results. Give it NULL instead, and it will advance.

That is really not how you're supposed to be using it. I repeat, this is how it's supposed to work:

Code:
char original_string[]="this is a string";
char *sep=" ";
char *token=strtok_r(original_string, sep, &tmp); // Give it the original string once and only once.

while(token != NULL)
{
        printf("token is '%s'\n", token);
        token=strtok_r(NULL, sep, &tmp); // Give it NULL.  Not the previous token.  Not the original string.  ONLY NULL!
}


Last edited by Corona688; 01-16-2014 at 04:13 PM..
This User Gave Thanks to Corona688 For This Post:
# 6  
Old 01-16-2014
Yes, your code worked perfectly and much simpler. The only thing bugs me is the NULL for the second strtok_r():
Quote:
word=strtok_r(string1, sep, &tmp1);
word=strtok_r(NULL, sep, &tmp1); // NULL means 'continue from last token'.
Swallowing it without digestion bothers me a lot. I was thinking this way according to the explanation:
Quote:
char *strtok_r(char *restrict s, const char *restrict sep, char **restrict lasts);
In the first call to strtok_r(), s points to a null-terminated string, sep to a null-terminated string of separator characters, and the value pointed to by lasts is ignored. The strtok_r() function shall return a pointer to the first character of the first token, write a null character into s immediately following the returned token, and update the pointer to which lasts points. In subsequent calls, s is a NULL pointer and lasts shall be unchanged from the previous call so that subsequent calls shall move through the string s, returning successive tokens until no tokens remain. The separator string sep may be different from call to call. When no token remains in s, a NULL pointer shall be returned.
:
Code:
word=strtok_r(ptr, sep, &tmp1))    //pop one token from the beginning, rest (&tmp1) is the left_over
ptr = tmp1;                      //move the pointer to the beginning of the rest (&tmp1)

But not sure, especially without the new pointer ptr. It seems I have to just remember and use it in your way. Trying more to get the prototype of strtok_r(), and to understand reentrancy, a related concept new to me. I hate C but I love it more.
Ok, MUST follow the usage of the function as migurus said:
Code:
...... on the first  call the 1st argument should point to the input string, then on the  following calls it (1st argument) should be NULL.


Last edited by yifangt; 01-16-2014 at 05:50 PM..
# 7  
Old 01-16-2014
Quote:
Originally Posted by yifangt
Yes, your code worked perfectly and much simpler. The only thing bugs me is the NULL for the second strtok_r()
From man strtok:

Code:
DESCRIPTION
       The strtok() function parses a string into a sequence  of  tokens.   On
       the  first call to strtok() the string to be parsed should be specified
       in str.  In each subsequent call that should parse the same string, str
       should be NULL.

strtok() wants a NULL because somewhere inside, there is an if(string==NULL) { // Use the string we had last time and literally no other reason. It's just a weird old library call that insists you use it in a very particular way.

There is no point worrying what the contents of tmp1 are either. It might not even be the same in a different libc.

I think your confusion is related to the concept of re-entrancy. A re-entrant function, if you call it twice with the exact same parameters, would do the exact same thing. strtok() violates this, because it remembers what string you gave it last time you called it.

Imagine you're breaking up the string "a:b:c|d:e:f|g:h:i" with strtok upon "|". It gives you "a:b:c", and you call strtok() again with ":" to break it into "a", "b", "c". In doing so, strtok() will forget the original string!

This is perfectly okay with strtok_r since you can give them different tmp variables. Those variables, not the function itself, will remember where it was last, so there is no conflict.

Last edited by Corona688; 01-16-2014 at 05:23 PM..
This User Gave Thanks to Corona688 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Function - Make your function return an exit status

Hi All, Good Day, seeking for your assistance on how to not perform my 2nd, 3rd,4th etc.. function if my 1st function is in else condition. #Body function1() { if then echo "exist" else echo "not exist" } #if not exist in function1 my all other function will not proceed.... (4 Replies)
Discussion started by: meister29
4 Replies

2. UNIX for Beginners Questions & Answers

Idea to make a function as microservice

Hello - I wrote few scripts on bash shell script and grafana triggers those scripts and show on console . I want to write the console output to a log file as well by using tee command and I am successful as well . I am wondering Instead of writing same logic on multiple scripts , why... (4 Replies)
Discussion started by: Varja
4 Replies

3. Shell Programming and Scripting

How to make nested function local?

Hi, If I declare a function inside another function, it overwrites any previously declared function with the same name. This is NOT what I want. Example: #!/bin/bash _test() { echo test; } _myf() { # I'm using the same name as the other function. _test() { echo local test; }... (8 Replies)
Discussion started by: chebarbudo
8 Replies

4. Shell Programming and Scripting

Is it possible make the shell read functions 1 by 1 and calling an other function?

Greetings, I m wondering if it's possible do do the following : I have a simple function called "FindMoveDelete" which does the following : FindMoveDelete() { find . -iname "$FILENAME*.ext" -exec mv {} "$PATH/$VAR" \; && find . -maxdepth 1 -type d -iname "$FILENAME*" -exec rm -rf {}... (6 Replies)
Discussion started by: Sekullos
6 Replies

5. UNIX for Dummies Questions & Answers

make - foreach function

I wrote the following Makefile: dirs := a b c d files := $(foreach dir,$(dirs),$(wildcard $(dir)/*)) .PHONY: all all: touch $(files) The first two lines are taken from GNU make tutorial, Section 8.5 The foreach Function. I would expect the recipe touch $(files) to be... (2 Replies)
Discussion started by: ybelenky
2 Replies

6. UNIX for Dummies Questions & Answers

Difference between configure/make/make install.

Hi, While installation of apache on linux, we perform the below tasks. 1) Untar 2) configure 3) make 4) make install. I wanted to understand the difference and working of configure/make/make install. Can any one help me understanding this? Thanks in advance. (1 Reply)
Discussion started by: praveen_b744
1 Replies

7. Homework & Coursework Questions

I need to make a script that has 4 command line arguments to function properly.

I have no idea what the following means. The teacher is too advanced for me to understand fully. We literally went from running a few commands over the last few months to starting shell scripting. I am not a programmer, I am more hardware oriented. I wish I knew what this question was asking... (3 Replies)
Discussion started by: Wookard
3 Replies

8. Solaris

Gani Network Driver Won't Install - make: Fatal error: Don't know how to make targ...

I attached a README file that I will refer to. I successfully completed everything in the README file until step 4. # pwd /gani/gani-2.4.4 # ls COPYING Makefile.macros gem.c Makefile Makefile.sparc_gcc gem.h Makefile.amd64_gcc ... (1 Reply)
Discussion started by: Bradj47
1 Replies

9. Programming

How to make a function friend to both base and derived class

Hi, I have a base class and derived a class from the base class, i want to print & read the data for the object created for the derived class,so i have overloaded both the << and >> operators and also have done the foward declaration. Below is the code snippet, #include <iostream> class... (3 Replies)
Discussion started by: ennstate
3 Replies

10. Shell Programming and Scripting

How to make variables in script function local?

Is it possible to make function variables local? I mean for example, I have a script variable 'name' and in function I have declared variable 'name' I need to have script's 'name' have the same value as it was before calling the function with the same declaration. The way to preserve a... (5 Replies)
Discussion started by: alex_5161
5 Replies
Login or Register to Ask a Question