It definitely works in W(ind) mode. What it is doing is finding the wind angle that will make the boat reach an imaginary waypoint directly to windward, as fast as possible. The VMG it is optimizing is the VMG to windward, NOT the VMG to the next navigation waypoint.
I know the manual gives very little to this function, but what it says (in total!) is "You can optimize the VMG to the wind. When selected the function will be active for 5-10 minutes after a new wind angle has been set and only when beating."
The assumption here is that if you are beating, then your desired destination must be to windward of your location. It actually doesn't matter for the optimization if that destination is directly to windward, as long as it is somewhere in the cone between the boat's tacking angles.
Note that the optimization in not continuous. It works until it is finds an optimum that meets its criteria (what ever that is!), and then holds that wind angle. If conditions change, such as wind strength or wave size, it is a good idea to command the unit to make a small change in course so it re-runs the optimization program.
As I said, I rarely use Wn (Wind Navigation) mode on my system, so I have little experience with how the VMG Optimization function works there, but based on what I understand about the Wn mode works, the VMG function wouldn't actually do anything
The optimized wind angle will always produce a speed through the water a bit slower than you could get if you bore off just a few degrees. Using the simplest possible analysis, if you are sailing 6 knots at a True Wind Angle of 55 degrees, anything better than 5.5 knots will give a better VMG at 49 degrees TWA
I do not know if this function uses only speed through the water and true wind angle for it's calculation, or if it adds in course and speed over ground. I emailed B&G's tech support asking that question, but never got an answer.