Windows 98/SE and ME had known issues with memory configurations over 512MB. Add the following line to the [VCache] section of your win.ini file: MaxFileCache = 524288.
I don't honestly remember if MSCONFIG works in 98. If it does, you can do it there. If MSCONFIG doesn't work, use SYSEDIT. This will allow you to add the line to your win.ini file. Just make sure that you do it under the [VCache] heading, and not just anywhere in the file.