Some vim installations have a pretty annoying default setting for mouse behavior to automatically enable visual mode.
To temporarily disable the behavior, especially to copy something off the terminal, issue
:set mouse-=a
To disable the automatic visual mode permanently, add the following to your ~/.vimrc
set mouse-=a