The grounds were originally the property of the Echigo Takada Clan and were the site of their Edo clan residence. The property later passed into the hands of the Maizuru Prefecture Governor, Makino Sukeshige, and finally became the official residence of the Iwasaki family. Over the years, there were more than 20 buildings on this 49,500m2 property.
At the end of World War II, the Iwasaki home and grounds became the property of the Japanese government and were used for the Judicial Research and Training Institute of the Supreme Court. In 2001, the City of Tokyo took on the responsibility for the management of the grounds and buildings.
In 1961, the western-style residence and the billiards building were designated important cultural assets, and in 1994 management of the grounds was turned over to the Agency for Cultural Affairs.
The Great Hall of Japanese-style building and the Japanese decorative screens in the East Wing of the western-style building were both designated important cultural assets in 1969. Later, in 1999, the land upon which the residence is located, and the surrounding tiled-walls were also designated important cultural assets.