Perhaps you should pursue this directly with Embest?
Stick a scope on the VGA pins and confirm the appropriate signalling, or perhaps find a monitor that expresses signal parameters on the screen, even for what it considers to be illegal modes. Different monitors have different capabilities in this regard, find one with suitable diagnostics will help with experimenting in this area.
Check that the VSYNC/HSYNC have appropriate polarities, and frequencies.
Check it with an older CRT display (support video bandwidths in the 200-300 MHz range), most LCD's are quite inflexible and have very low refresh rates in ~60 Hz, many VGA modes use 85 Hz, most consumer LCD's are typically in the sub 100 MHz bandwidth range.
At the very least get a datasheet for the monitor you're trying to use, or extract the DDC data, or check the supported modes in Windows.
http://martin.hinner.info/vga/timing.html