I don't think the size of the file you are writing here has any bearing. If your OS is trying to switch a large file system call for your regular call because it thinks the file is to big then your OS is horribly broken. Not only would that be a stupid and useless thing to do, but the standard clearly says that the program should get an error.
Since you said the program is a 32 bit program and isn't supposed to handle large files, I'm going to assume that it doesn't have _LARGEFILE64_SOURCE defined.
That being the case, the only way I can see that your program can be calling open64 is by calling open and having the system somehow think it's supposed to be opening a large file. To do that you either have to use the O_LARGEFILE flag, or directly set _FILE_OFFSET_BITS to 64 or or have set a bit via FCNTL().
The only other option I can see is along the lines of your entire system entry point getting mangled. And if that where the case I would expect the entire system to go down with it.
Is it possible that something is closing the file and that is triggering an attempt to reopen the file when you next write to it? Unless it's hoplessly messing up your system then something must be doing an open call.
Jay