I think I figured out the problem.
The DC toy motor I purchased required a lot more current then others. Sparkfun sells 2 kind, one with just 70mA spec as no load current and the other one is 320ma no load current.
The circuit is not giving enough current into the bigger motor and it seems to work if I change the speed instantaneously instead of gradually changing it.
Thanks for all your guidance.
Cheers.