x86: don't ignore mandatory pseudo prefixes

x86: don't ignore mandatory pseudo prefixes

{vex}, {vex3}, and {evex} are mandatory prefixes, and hence should not
be randomly ignored. Fix this for insns without operands as well as for
insns referencing the high 16 [XYZ]MM registers. To achieve the former,
re-purpose VEX_check_operands(), renaming it to VEX_check_encoding() and
moving its only operand check to check_VecOperands().

This involves fixing a testcase relying on {vex2} to get ignored.