As someone who lived in the downtown/west end boundary, yes it's all considered downtown. (And a big chunk of West End is the park, anyway...) The map differentiates it for political/services reasons but apart from seeing a few less glass condos its pretty similar. West End has more in the way of trees and older houses, but it has been pretty rapidly condo-izing. About the only thing Downtown side has over it is the mall and train stations.
EDIT: The terms are not very useful anyway; each contains multiple smaller areas with distinct differences, ie: coal harbor versus robson vs. denman, or yaletown vs. chinatown vs. hastings.
Ahh. Like someone from Surrey saying they live in Vancouver
Minus a couple new developments, the experience of living in a West End condo and a downtown condo is huge. I mean we just had a conversation over bed bugs. That's not an issue when you are living in the buildings downtown. You don't need "laundry mats" in your buildings downtown. Almost everything in the west end is run down and stuck in the 70s/80s. The only part of downtown that has any semi-old buildings is the corner of Coal Harbour (which is more on the West End side) and Gastown (but a lot of that is due to historical preservation and a lot of that is changing with gentrification)
Also that differentiation in services/political reasons is important. There's more money dedicated to services in Downtown.
As far as Vancouver goes, this is the order of economic status: Point Grey > Arbutus / Shaugnessy > Downtown Vancouver > Kits > Fairview (with the amount of money False Creek is getting) > West End. The average difference in socio economic status between those areas is a noticeable difference.