Shared mem where? In the database?
Where is the Java code running? In the database? Out of the db, but on the same box? A different box?
If it's the shared memory in the database, and the Java process is not a Java stored proc, then it's not the Java running out of memory (sorry, Ross, you're full of crap :-). Oracle requires a lot of shared memory to run properly - it's used for many different things. Java may be trying to create a prepared statement on the fly, and Oracle is running out of memory trying to parse it because too many other things are using the shared memory at the moment.
Oracle recommends using half of the available RAM on the box for shared memory.
You might try pmap (if this is on Solaris) to figure out where it's going.