I'm not so worried about somebody building their own nukes as somebody acquiring somebody else's nukes *COUGH*Russia*COUGH* and using them on a city or port.
In Clancy's "The Sum Of All Fears", the terrorists started with an Isreali nuclear missle (and hired a Russian scientist, I believe) and the level of effort to turn it into a bomb was *still* extraordinary (and Clancy didn't include everything)
The precision equipment needed is very expensive and very hard to get a hold of. The expertise to do it right is rare. Creating the working conditions is prohibitive.
Starting with a nuclear missle and building a bomb is like starting with just a computer chip and building an entirely functional PC. The general theory is easy, the details are daunting. You need to know a lot of specific information and get your hands on specific parts and equipment; and those specifics are not easy to come by with nuclear weapons