Where do I look to buy a bike? What costs should I expect?
If you're unsure, go to a local bike shop (LBS). Bike shops are the most expensive option for getting a given bicycle in your hands, but:
1-You can try out different things on the spot.
2-You'll get fitted.
3-They'll service any early problems free of charge, and the purchase typically comes with one tune-up for correcting for the breaking-in process.
Lowest-end LBS bikes typically fall in the $300-$400 range. At this price point you can get heavy beach cruisers and super-upright hybrid/mountain-bike/cruiser thingies. Doesn't take much more than that to get a perfectly good single-speed bike. And for double that you can get a reasonable, if basic, multi-speed entry-level road racing bike.
Diminishing returns set in quickly. For $15,000 you can get an 11-pound 22-speed fancy-schmancy carbon-fiber high-performance machine that feels tighter and will get you to the top of a hill a few seconds faster.
In addition to the bicycle itself, people often wind up wanting to change saddle for fit reasons. Basic replacement saddles could run $30-$60 or so.
Most entry-level bikes come with cheapo platform pedals. These more or less do the job, and plenty of people are fine with them.
Some people wind up wanting grippier platform pedals. There are various options, notably pedals that have set screws sticking out of them that dig into your sole. These will also draw blood if any part of you so much as grazes said set screws, but that's part of the fun! Expect to lose another $40-$100 dollars if you decide to go down this route.
Some people wind up wanting to use "clipless" pedals (which do the opposite of what the name sounds), where you have a cleat on your shoe that clips into the pedal, securing your foot to the cranks. This is nice in that, if the cleat has been positioned correctly on your shoe, you can ride your bike without thinking about foot contact or positioning. If you decide to go down this route, you might be looking at something like $30-$120 for the pedals+cleats and $50-$120 for the shoe. The downside of these is that you're using a cycling-specific shoe: there are "mountain bike" pedal options like SPD where the cleat sits into the shoe and walking feels mostly normal-ish, but there are also "road bike" pedal options like SPD-SL where the cleat sticks out from the shoe and walking is a bit clunky. So this needs to be kept in mind for commuting (i.e. do you carry an extra pair of shoes with you to work).
Other things:
Bottle cages could run you $10-$20 each depending on whether you get basic metal ones or fancy-schmancy nylon ones.
You should get some manner of repair kit, enough to adjust basic things on your bike and repair flats. This means a multitool with at least screwdrivers and hex keys, tire levers, a spare tube, tube patches, a saddle bag to shove this stuff in, and either a pump or a CO2 setup. Some LBSs sell basic kits containing basically all of this for around $50. If you want to pick out fancier tools and a higher-quality saddle bag, this can quickly run you over $100.
What kind of bike should I consider?
Depends on what you care about and what you think you might wind up doing with it.
If your city doesn't have have lots of steep hills and you want a machine that's simple and reliable and requires minimal maintenance, an urban single-speed might be your thing.
What should I know about city riding? Is it even a good idea?
It can be intimidating, depending on where you're going and in what city. You'll want to learn your roads. But it's good exercise and, depending on when you're riding, can be faster than taking a car.