x86: refine when to trigger optimizations

x86: refine when to trigger optimizations

Checking just the base opcode without also checking this isn't a VEX
encoding, and without there being other insn properties avoiding a match
once respective VEX/XOP/EXEX-encoded insns would appear, is at least
dangerous. Add respective checks. At the same time there's no real need
to check the extension opcode to be None for the 0xA8 form - there's
nothing it can be confused with, and non-VEX-and-alike forms also can't
appear.