When you push new data onto an array, Perl has to periodically grow the underlying data structure (which involves some copying). A lot of data pushed means a lot of growing/copying. With a hash, you're still growing the underlying data structure, but the impact isn't as bad, depending on the mix of keys and how they hash.
You can preallocate the array by doing something like
my @array = (undef) x 100_000;
and then assign into it by index. That'll eliminate the overhead you're seeing from push().