Relax standardness rules regarding CHECKMULTISIG

In PR bitcoin#5247, the STRICTENC standardness rules were tightened with regards to
CHECKMULTISIG so that unparsable public keys fail the script when they are
encountered. The overall purpose here was to disallow the use of confusing
hybrid public keys by policy while keeping policy compatible (i.e., strictly
stronger) with consensus rules.

Comments in PR bitcoin#5247 note that "I don't believe it should affect any system in
production"; however, this belief is/was false. Counterparty was stuffing data
blobs into multisig pubkey lists, but these UTXOs were meant to be spendable
because, although some pubkeys were unparsable, other keys were parsable, and
the UTXOs were meant to be spent by those valid keys.

But in tackling the hybrid key issue, PR bitcoin#5247 disallowed any unparsable keys in
multisigs, whether or not they were hybrid, and whether or not the signature was
meant to satisfy a hybrid key.

In production, Counterparty UTXOs were inadvertently caught up in this
standardness rule change and became "soft confiscated." That is, they were no
longer spendable by policy but still recoverable if users are able to somehow
bypass standardness by mining their transactions themselves, or getting
out-of-band assistance from some other miner.

I understand that Bitcoin Core never intended to "soft confiscate" any UTXOs by
policy changes. This change addresses the problem.

With this change, standardness rules intended for hybrid keys are only checked
after passing a signature check in CHECKMULTISIG operations. Failing signature
checks revert to their consensus behaviour of testing subsequent public keys.

Counterparty UTXOs were never intended to make use of hybrid keys and thus
shouldn't have any passing signatures using hybrid keys.

Author: roconnor-blockstream

src/script/interpreter.cpp

@@ -1158,7 +1158,7 @@ bool EvalScript(std::vector<std::vector<unsigned char> >& stack, const CScript&
                         // Note how this makes the exact order of pubkey/signature evaluation
                         // distinguishable by CHECKMULTISIG NOT if the STRICTENC flag is set.
                         // See the script_(in)valid tests for details.
-                        if (!CheckSignatureEncoding(vchSig, flags, serror) || !CheckPubKeyEncoding(vchPubKey, flags, sigversion, serror)) {
+                        if (!CheckSignatureEncoding(vchSig, flags, serror)) {
                             // serror is set
                             return false;
                         }
@@ -1167,6 +1167,10 @@ bool EvalScript(std::vector<std::vector<unsigned char> >& stack, const CScript&
                         bool fOk = checker.CheckECDSASignature(vchSig, vchPubKey, scriptCode, sigversion);
 
                         if (fOk) {
+                            if (!CheckPubKeyEncoding(vchPubKey, flags, sigversion, serror)) {
+                                // serror is set
+                                return false;
+                            }
                             isig++;
                             nSigsCount--;
                         }