NYC grew (and shrunk) in fits and starts. As the previous poster stated, NYC originally consisted of only Manhattan Island.
In 1874 the western portion of the present-day Bronx (the area west of the Bronx River) was annexed by NYC from lower Westchester County; it was unceremoniously called The Annexed District, or the North Side.
In 1895 the eastern portion was annexed.
In 1898 NYC was enlarged to include all of Staten Island (Richmond Co.), all of the City of Brooklyn (Kings Co.) and the eastern third of Queens Co. (which at the time extended to the Suffolk Co. Line) including the city of Long Island City. This greatly-enlarged municipality was divided into 5 boroughs, and was commonly called "Greater New York" (though the legal name remained simply the City of New York).
In 1899 Nassau Co. was created from the eastern 2/3 of Queens Co. that had not been included in Queens Borough. In the process, a little bit of territory (about 20 sq. miles) was taken back from Queens Borough and given over to Nassau Co.
These days, the term "Greater New York" is loosely applied to the 5 boroughs plus the surrounding suburbs -- LI, Westchester, even NJ. It is an informal term used to describe the vicinity, rather than the formal boundaries, of today's 5-borough city.