Initially, The Esplanade was a pathway along the waterfront. Part of the Walks and Gardens. Then the railways were allowed to bring in rail lines south of it and it became, for all intents and purposes, a rail yard. I have never heard that it was paved in stone and not at all sure that it's heyday was in the 19th century. If anything, it became more interesting only in the early 20th century with some of the buildings still on north side between Church and Yonge.
The Esplanade was actually authorised only in 1852/1853 and again in 1857. See:
https://bnald.lib.unb.ca/sites/default/files/UnC.1852.ch_.219.pdf and
https://bnald.lib.unb.ca/sites/default/files/UnC.1857.ch_.80_0.pdf
Laws of Her Majesty's Province of United Canada, passed in the year 1852. 16 Victoria – Chapter 219
An Act conveying to the City of Toronto certain Water Lots, with power to the said City for the construction of an Esplanade. Assented to 14th June, 1853. Whereas by Letters Patent, under the Great Seal of the Province of Upper Canada, bearing date the twenty-first day of February, in the year of our Lord, one thousand eight hundred and forty, certain water lots or tracts of land covered with water, situate in front of the said City of Toronto, and certain parcels or slips of land situated between the top of the bank and the water’s edge of the Bay, in the said City of Toronto, adjoining to the said water lots, were under the direction of an Order in Council of the 17th August, 1837, granted to the Mayor, Aldermen and Commonalty of the said City of Toronto, and their successors for ever, upon trust, to lease the said water lots, or apply them to and for the public purposes of the said City, as the Common Council of the said City of Toronto, from time to time, might think fit to order or direct:
And:
An Act to amend the-Act conveying to the City of Toronto certain Water Lots, with power to the said City for the construction of an Esplanade, and to enable the said City to locate the Grand Trunk Railroad and other Railroads along the frontage of the said City. Assented to 10th June, 1857.
It used to go west of Yonge, until Union Station was built but even in 1890 Goads, there was very little development and virtually none on south side.
or looking towards the Distillery from about Jarvis: