I pretty much refuse to play 3v3 because for some reason I never win there, so in regards to 5v5 as Riven you'll want to be solo-top most of the time (presuming you have a jungler, which you should if you're over level 16, otherwise going duo top lane as Riven is just as fine).
You'll want to make sure you can always buy a green vision ward on every trip back to base and then place that ward in the river near your brush, this will let you see if any enemy players are trying to come in and gank you.
If your opponent is a another melee bruiser like Irelia, Nasus, etc, then you'll want to be trying to do as much harass as you can if you appear to be the stronger player & champion, but you need to make sure you don't over extend too much (being in their half of the lane for too long), if you've got your lane pushed up to their turret, take that chance to recall and spend your excess gold. Also, you can try playing slightly more passively and letting your minion waves get pushed back to your half of the lane, which will force the enemy champion to either over extend or miss out on farm at which point you should eventually be able to catch him out of position and try and go for the kill.
Like I said, at the end of the day it comes down to practice. The more you practice with the same champion in similar scenarios (i.e. solo top Riven) the quicker you'll pick up subtle 'knacks' and tricks to playing that set up.