Parallelism, and how the controller makes use of it is the main factor. There usually isn't room for the 8-16 discrete NAND chips you'd find in an SSD. The controllers in USB stick usually aren't nearly as complex either, to efficiently make use of available parallelism.
Other factors that matter are the quality of the flash. Many USB sticks use cheaper flash which you have to read and write slower in order to avoid errors. USB drives don't have lots of over-provisioning to help keep clean blocks around. And USB 2.0 is limited to around 35 MB/s. And many SSDs come with a big RAM cache.
The main reason for these differences is price. There are some SSDs in a USB stick form factor, like the LaCie FastKey.