Sponsored Content
Full Discussion: Tokenization
Top Forums Programming Tokenization Post 302136963 by porter on Thursday 20th of September 2007 05:45:29 PM
Old 09-20-2007
strtok - C++ Reference

My understanding is you provide the string and a list of delimiters (commas, spaces etc)

Code:
char *reply_ptr=reply;
char *token=NULL;

do
{
     token=strtok(reply_ptr,";. \r\n");

     reply_ptr=NULL;

     if (token) { .... }

} while (token);

 
strtok(3)						     Library Functions Manual							 strtok(3)

NAME
strtok, strtok_r - Split string into tokens LIBRARY
Standard C Library (libc.so, libc.a) SYNOPSIS
#include <string.h> char *strtok( char *s1, const char *s2); char *strtok_r( char *s1, const char *s2, char **savept); STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: strtok_r(): POSIX.1c strtok(): XPG4, XPG4-UNIX Refer to the standards(5) reference page for more information about industry standards and associated tags. PARAMETERS
Contains a pointer to the string to be searched. Contains a pointer to the string of byte token delimiters. [POSIX] Identifies the loca- tion of the byte where the search for tokens should be started in the next call to strtok_r(). The savept parameter contains a pointer to a variable that contains a pointer to the byte in the string. DESCRIPTION
The strtok() function splits the string pointed to by the s1 parameter into a sequence of tokens, each of which is delimited by a byte equal to one of the bytes in the s2 parameter. Usually, the strtok() function is called repeatedly to extract the tokens in a string. The first time the application program calls the strtok() function, it sets the s1 parameter to point to the input string. The function returns a pointer to the first token. Then the application program calls the function again with the s1 parameter set to the null pointer. This call returns a pointer to the next token in the string. The application program repeats the call to strtok() with the s1 parameter set to the null pointer until all the tokens in the string have been returned. Note If the input string contains no instances of bytes from the delimiter string, the first call to strtok() results in the return of a pointer to the original string. On subsequent calls, strtok() returns NULL. In the initial call to strtok(), the function first searches the string pointed to by the s1 parameter to locate the first byte that does not occur in the delimiter string pointed to by the s2 parameter. If such a byte is found, it is the start of the first token. The str- tok() function then searches from there for a byte that does occur in the delimiter string. If such a delimiter is found, strtok() over- writes it with a null byte, which terminates the current token. The strtok() function saves a pointer to the byte following the null byte and returns a pointer to the start of the token. In the subsequent calls to strtok(), in which the s1 parameter is set to the null pointer, the function starts at its saved pointer and searches for the next byte that does not occur in the delimiter string pointed to by the s2 parameter. If such a byte is found, it is the start of the new token. The strtok() function then searches from there for a byte that does occur in the delimiter string. If such a delimiter is found, strtok() overwrites it with a null byte, which terminates the new token. The strtok() function saves a pointer to the byte following the null byte and returns a pointer to the start of the new token. If a call to the strtok() function cannot find a byte that does not occur in the delimiter string, it returns the null pointer. If a call to the strtok() function cannot find the terminating byte that does occur in the delimiter string, the current token extends to the end of the string and subsequent calls to strtok() will return the null pointer. If the delimiters used in the string change from one set of characters to another within the string, the application program can set the second parameter, s2, to different strings from call to call. The implementation behaves as though no function calls the strtok() function. The strtok_r() function is the reentrant version of strtok(). Upon successful completion, the strtok_r() function stores the saved pointer in *savept. If the s1 parameter is a null pointer, the strtok_r() function uses the saved pointer in *savept to start searching for the next token. In the initial call to strtok_r(), the *savept must be the null pointer. NOTES
[POSIX] The strtok() function is not supported for multithreaded applications. Instead, its reentrant equivalent, strtok_r(), should be used with multiple threads. EXAMPLES
The following example demonstrates how to split a string into tokens. #include <string.h> #include <locale.h> #include <stdio.h> #define LENGTH 40 main() { char string1[LENGTH], delimiters[LENGTH]; char *pstr ; int counter; (void)setlocale(LC_ALL, ""); printf("Enter the string to be searched: "); if (fgets(string1, LENGTH, stdin) != NULL) { printf("Enter the delimiter(s): "); if (fgets(delimiters, LENGTH, stdin) != NULL) { if ((pstr = strtok(string1, delimiters )) != NULL) { /* pstr points to the first token */ printf("Token 1 is %s ", pstr); counter = 2; while ((pstr = strtok((char *)NULL, delimiters )) != NULL) { printf("Token %d is %s ", counter, pstr); counter++; } } } } } RETURN VALUES
Upon successful completion, the strtok() and strtok_r() functions return a pointer to the first byte of the parsed token in the string. When there is no token in the string, a null pointer is returned. RELATED INFORMATION
Functions: string(3), wcstok(3), wcstok_r(3) Standards: standards(5) delim off strtok(3)
All times are GMT -4. The time now is 11:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy