The three "opened the door" to an unknown world where matter takes unusual states or phases, the Royal Swedish Academy of Sciences said.
They were for their "theoretical discoveries of topological phase transitions and topological phases of matter."
Thouless, 82, is a professor emeritus at the University of Washington. Haldane, 65, is a physics professor at Princeton University in New Jersey. Kosterlitz, 73, is a physics professor at Brown University in Providence, Rhode Island.
Their research was conducted in the 1970s and '80s. Nobel judges often award discoveries made decades ago, to make sure they withstand the test of time.
This year's Nobel Prize announcements started Monday with the medicine award going to Japanese biologist Yoshinori Ohsumi for discoveries on autophagy, the process by which a cell breaks down and recycles content.
The chemistry prize will be announced on Wednesday and the Nobel Peace Prize on Friday. The economics and literature awards will be announced next week.