House of Mansions to Bankers' Trust

This post also appears on the Bryant park blog.
 
In 1855, developer George Higgins bought a plot of land at Fifth Avenue and 42nd Street, across the street from the Croton Distributing Reservoir. He hired American architect Alexander J. Davis to design a new building, and the result was this 11 unit luxury residential complex with crenelated parapets (like a castle!). Completed in 1856 and advertised as the House of Mansions, the housing complex promised residents views of "the water of the Croton, like an artificial pool, or lake . . . from the upper floors."  

Fifth Avenue looking south from 42nd Street. Image [1880]: NYPL or LOC

Just a few years earlier, the nearby Crystal Palace and Latting Observatory attracted visitors from all over, and helped establish midtown Manhattan as a tourist destination. Though the area was far from urban at the time, Higgins no doubt noticed the gradual migration of wealthy Manhattanites north along Fifth Avenue, and hoped to cash in on it. 

As history blogger Daytonian in Manhattan writes, Higgins' plan did not work -- Fifth Avenue at 42nd Street was still considered too rural for most New Yorkers. (The  Samuel P. "Sasparilla" Townsend residence had only recently been built at 34th and Fifth, and that's eight blocks south.) The property was bought and sold a few more times, and eventually the Rutgers Female Institute (later, Rutgers Female College) relocated here. A formal dedication ceremony was held on October 24, 1860. The school soon outgrew this location as well, and the property passed through a few more owners.

Fifth Avenue looking north from 41st Street, [1883].

By 1884 the Pottier & Stymus Manufacturing Company had purchased the land, demolished the structures, and built a new building on the site. The New York Times called it "extensive and elegant," and described the building as having a brownstone frontage on Fifth Avenue as well as a side entrance on 42nd Street. (I think it's the building in the photo below with the wall sign that reads "American Safe Deposit Bank.")

Fifth Avenue, looking north toward 42nd Street [1900-1910]. Image: LOC

By this time the landscape of Fifth Avenue had changed from rural to commercial, with a brief residential heydey in between. The area surrounding 34th Street by now had transformed from residential to retail, and many of the brownstones near 42nd Street were rapidly being torn down and replaced with commercial buildings and warehouses. 

The next ten years brought more changes and taller buildings to Fifth Avenue. In 1915 the Oceanic Investment Company announced the construction of a new building to be named after the building's primary tenant, the Astor Trust Company. The Astor Trust Company was set to move from their existing offices on Fifth Avenue and 36th Street and into this building, occupying a 21-year lease upon the building's completion in 1917.

Astor Trust Company building, NE corner of Fifth Ave. and 42nd Street, 1917. Image: LOC

The building's architect was Montague Flagg, also known for designing the Thomas Cook building at 565 Fifth Avenue, which is no longer. The Astor Trust Company building was eventually renamed the Bankers' Trust building and still stands, though it's no longer the tallest on the block, and there isn't a crenelated parapet to be found. 

Other Sources:
The Fifth Avenue Association. Fifty Years on Fifth, 1907-1957.
Lockwood, Charles. Manhattan Moves Uptown: An Illustrated History.