...yep...buy a switch , or do it manually.
There isn't an easy way I'm afraid. The only other way would be to use a stereo replicator, or a 'splitter'- so that you can hook up both your speaker's 3.5mm audio jack and headest 3.5mm audio jack into the speaker port. However, most of these also split the quality of the audio...so they aren't a good solution. Unless, however, you can find a splitter which does not lose sound quality .