Agree, but...
The implication here is that if I work in a government building, I must surrender my right to worship/free speech. Since it's a government building, I cannot display any religious artifacts in my cube/office/desk. By same implication, I cannot have/read my bible at my desk during a break, because that would imply government sponsored religion.
Government cannot tell me who/what to worship, cannot force me to pray, cannot decide that the "Holy Order of the Shrub" is the one true way. The government should not be telling me NOT to pray/worship either.