You can (and people have) Open Source Smalltalk, just that your code is targetted for a particular version of Smalltalk. You can open source your VisualWorks code and anyone can use it in VisualWorks applications, but it probably won't work in Dolphin without modification

Squeak itself is Open Source, but I believe it's the only IDE/compiler in that status