regcomp: recognize ill-formed { } expressions