Let's get the simple things out of the way first.

Transactions don't work for MyISAM tables; they have to be InnoDB. You knew that already, didn't you? :-) And then there's MySQL's AutoCommit flag, although START TRANSACTION is supposed to disable that until you do COMMIT (or ROLLBACK). But it's probably the table-type.

The problem with the query is that you're referencing the same table in the sub-query that you're trying to update. MySQL doesn't allow that. Why can't you shift the sub-query's condition to the outer UPDATE?

UPDATE queue
SET status = 'process-$PID'
WHERE status = 'pending' AND target_time > NOW()
ORDER BY target_time
LIMIT $batch_size

That's atomic, even on MyISAM tables.

Wade.