The StringTokenizer uses a set of charactrers as delimiters, whereas The Scanner and the String class's split method uses the regular expression as delimiter.
The Scanner is rich in teh regular expressions which it uses for testing and then getting the next token. eg. we have the hasNextInt method which can test whether the next token identified using the delimiter is a valid int or not. This kind of testing before testing is not available with the other two options.
There is another major difference between StringTokenizer and the other two options.
Say we have a String s = "abc,xyz,mno,,pqr", and we use ',' as a delimiter then the StringTokenizer would give four tokens, whereas the other two would give us five tokens, the secondlast token would be a null. The Scanner class also has advantage of working directly on a character stream.
Hope this is helpful in understanding the difference between Scanner and StringTokenizer.



