What does the string tokenizer have to encapsulate?
The easiest thing is to separate it entirely from a specific input method (this way it is reusable) and therefore not responsible for its own file i/o methods.
You would ideally want to make a class (C++?, Java?) and pass it a string in the constructor. You would also want to pass a string containing string delimeter arguments....
for instance...
blah = new myStringTokenizer(myString, "\n\t ");
This would delimit strings by newline (\n), tab (\t) and whitespace( ).
Your class would ideally save the string to a member variable. Then it would save the delimiter string also.
You then want to call blah.nextToken() to return the next string token. Inside your class, you store the current location of your 'pointer', to remember where you left off, and incrementally build a string using a concatenation operation or another method until you encounter a delimeter character. Then you return said string.
[Another method if concatenation is not available (depending on what libraries you are using for string) is to store the start pointer, and move a working pointer along the string until you reach the delimeter char. Then you mark that position. You can then either call some variation of substring() or mid() or you can even then dynamically create the *correctly* sized string and loop back through from start pointer to end pointer copying char by char. This is much more efficient than concatenation]
Once delimeters are reached, you simply increment your position pointer along the string until a non-delimeter char is reached.
I would recommend having a member function/method called charisDelim which returns 1 if the given char matches a char in the delimeter string, or 0 otherwise.
__________________
Desktop machine: 2 x Opteron 246, Asus K8N-DL, 2GB PC3200 ECC Reg., XFX GeForce 6600GT, 74gb WD Raptor, 2 x 19\" LCDs, Windows XP x64
Server machine: Intel P4 3.0GHz 2MB EM64T, ECS i865pe, 1GB PC3200, 36gb WD Raptor, Windows Server 2003
Laptop: Dell Inspiron 9100 (Intel P4 3.2GHz 1MB Prescott, i865pe, 512MB PC3200, Mobility Radeon 9700, DVD+R/DL Burner), Windows XP
Linux: P3 450Mhz, 386MB ram, Slackware 10.1 (Running mySQL/Apache) | 
Linear Mode
