With high real estate prices in Vancouver and it not being a big business centre, there's not a lot of disposable income around. In Vancouver, the high end is mostly for tourists, esp. cruise ship passengers, so a number of high end shops are relatively close to Canada Place.
It will be interesting to see what happens with Holt's new Vancouver store - the old location mid-mall caught walk-by traffic (i.e. on the way to the food fair). Now it's at the end of the mall (across the street connected by a bridge and tunnel, like The Bay (Simpsons) at Eaton Centre in TO) and unless you're actually going there as a destination, you don't even think about it. The food fair is now mid-mall, so many business people from the Pacific Centre office towers don't even venture to that end of the mall at all. On top of that, the store is harder to negotiate because it is organized by designer, so if you're looking for a casual shirt, for example, you have to check 6 or 7 different locations in the store, whereas with the smaller store, everything was compact.