I got one as my first bike around March this year. Probably put around 4,000 miles on it so far (work,school,girlfriends blah.. I wish I could ride that much per month!)
The bike has been PERFECT for me. I rode dirtbikes when I was younger, and was already used to the controls on a bike. I also took the MSF course before I got my bike, the instructor actually recommended the SV for me, because he said I appeared very comfortable on a motorcycle.
My first impressions from riding the SV was more or less "HOLY SHIT THIS THING IS FAST." I still had a month before my permit expired so I couldn't get my full licence yet, so every day I'd practice in my Dad's neighborhood (which is pretty car-less for most of the day), and by the time I got out on the real roads I was quite comfortable with the power and engine braking.
My first couple times out on the street I'd get nervous at lights and stall it, or I'd let the clutch out a little too quick and the bike would jump forward and scare the crap out of me. However it NEVER lifted the front end on me without me wanting to (which came much later

). The bike is definately more demanding to ride than an EX500 or 250.. but I'd say it's much more rider-friendly than my friend's 636.
I'm very comfortable on it now. I can put my knee on the ground pretty easily, ride it pretty much every day to work, and most weekends I'll go for a really long ride around and find some twisty roads. I'm always still learning on it though, every time I got out I come back knowing somthing new about the bike or riding in general. The good thing about the SV(S) is that it takes a very, very long time to be able to ride this bike to it's limits.. and the aftermarket for the bike is huge, so you can upgrade everything on it if you ever feel like you need somthing more.
I'm now helping teach my friend how to ride, and he just got a 2003 SV650 (which is actually a little quicker than my 2001 SV650S).. He was really scared of the bike before he took his MSF course, but now after he finished the class, he comes on rides with me and is really getting the basics down. So I'd say, if you feel comfortable on the bikes at the MSF course, and know your limits on a motorcycle, I'd say that an SV or SVS would be a good starter bike.