Switches separate the traffic so each port sees only it's own traffic, not all traffic as with hubs (switches are just fast multi-port bridges).

If you have a gigabit leg between a server and a switch, that cable pools all traffic coming from all legs to that switch. If your switch had 10 100-Meg ports and one gigabit port to the server (which must, of course, have a gigabit card), you'd have to saturate all 10 100-Mbs ports with traffic for that server to saturate the gigabit leg, If the leg to the server is 100-Mbs it will be easily saturated and the legs out of the switch will be unable to achieve anything near 100-Mbs because there's a bottleneck at the server.

If you have peripheral switches feeding into a main switch with the servers on it, the legs to the peripheral switches must be gigabit also or you have serious bottlenecks between the switches.

My default would be a gigabit switch with the servers and peripheral switches connected to it all through gigabit ports, and the peripheral switches would have 100-Mbs ports facing the workstations. Of course, I'd try to push those peripheral switches out to the workgroups if possible to cut down the cost of pulling cable.

If all your switches are 100-Mbs on all ports, including the server ports, rewiring is highly unlikely to give you much of a performance boost.