Simple
If the person is a US citizen leaving the US TO go to another country, they are emigrating from the US. Using the United States as the frame of reference, they are emigrating. If you use the place they are moving to as the frame of reference, they are immigrating. So, if you're in New York, watching them sail past the Statue of Liberty in some human scow, you'd say they are emigrants. Some time later, in (for example) Sri Lanka, someone watching them sail in an the aforementioned human scow would call them immigrants (and probably be holding signs saying "Yankee Go Home" to boot).
You can think if it as emigration == uploading people (emigrants), and immigration == downloading people (immigrants).
If he wanted to discuss both the inflow and outflow of people, he should have simply used the word "migration", as in "permitting unrestricted migration to and from the US", or to be even more specific considering the context, "permitting unrestricted labour migration to and from the US". However, he was specifically talking about the movement of US labour "to ANY country in the world", which means he's talking about emigration, not immigration.
Mind you, the US already has this in certain labour sectors: Mexican fruit pickers in California and Florida already represent an unrestricted flow of seasonal labour over one of their borders. However, it's not strictly speaking legal (though it's entirely possible that the blind eye has been turned for so long that an argument could be made that it is in fact legal due to lack of enforcement).
Of course, this leaves the question of how the US would force target countries to accept US immigrants completely open...
