There's two philosophies for AP-esque Kayle builds:
ASPD/AP where you basically do as you outlined. You harass when your E is up, Q+E to orbwalk* them, and Q+W to either escape or pursue. Use your ult when they get on you** so you can take the brunt of their burst and keep trading. This is more used on champions that you can get away with multiple auto-attacks on; ones who have trouble disengaging you or trouble getting on top of you and sticking. Basically, the Nashor's Tooth + Liandry's build that lets you do more sustained damage so you need to be able to attack consistently.
Raw AP where you Q + E auto-attack and then disengage. Your harass is more on your Q dealing a lot of damage and your W heal + movement lets you just shrug off any damage you take and get out as fast as possible. Deathcap + Lichbane build to maximize the two-hit damage (Q+E for Q Damage, Auto Damage, Lichbane proc). This is more meant for champions you don't want to fight head-on or ones you can't sit and attack constantly.
Generally speaking, you can get away with AP/ASPD vs everyone, just with less of a kill potential in lane (e.g., trying to auto-attack down a Nidalee is pretty hard). There might be a few hard gap closers that you'll not want to but those would most likely beat raw AP too anyway :x
*Attack-move kiting where you Auto Attack, move to cancel the auto recovery frames, attack again as soon as your auto-attack "coolsdown", move to cancel the auto recovery frames, etc. etc. Basically, just running and attacking and minimizing how long you ever just "stand there" doing nothing.
**Since you mention Vi, I feel like noting that you can actually ult after she reaches you but before the knock-up so you can shield her ult damage and part of her follow-up damage.