Things to worry about: general street noise, not pleasant if you are near the garbage removal area, or loading area of the building, car pollution if near busy street, usually views are not great, etc.
Sometimes there's good things too. Some buildings (like mine) have "ground floor" penthouses with semi-private large backyards shared between two units only. Maybe also real patios instead of balconies. Elevators (if they break down, or they are busy) don't become an issue since you won't be spending much time in them, if you need them at all.