His earliest recorded large commission was in his capacity as a builder and timber merchant at Thrigby Hall in 1735.He then leased a piece of land on the west forecourt of the Great Hospital on which he built a substantial residence for himself, into which he moved in 1756.He went on to build the Assembly House (1754), the neighbouring theatre (1757, since destroyed),[3] the Methodist meeting-house in Bishopsgate Street (1752–3), [4] the Octagon Chapel in Colegate (1754–6) and the artillery barracks (1771).[3] Between 1767 and 1779, along with two members of his family, his son William and his nephew John Ivory,[5] he built a new range at Blickling Hall, closing in the open courtyard in a style sympathetic to the existing Jacobean house.[2] In addition to work as a builder and architect, Ivory carried on an extensive trade importing timber, with his own yard in Bishopsgate.