Relax standardness rules regarding CHECKMULTISIG

In PR bitcoin#5247, the STRICTENC standardness rules were tightned with regards to
CHECKMULTISIG so that unparsable public keys fail the script when they are
encountered. The overall purpose here was to disallow the use of confusing
hybrid public keys by policy while remaining keeping policy compatible (i.e.
strictly stronger) than consensus rules.

Comments in PR bitcoin#5247 note that "I don't believe it should affect any system in
production", however this believe is/was false. Counterparty was stuffing data
blobs into multisig pubkey lists.  But these UTXOs were meant to be spendable
because, although some pubkeys were unparsable, some keys were parsable, and the
UTXOs were meant to be spent buy those valid keys.

But in tackling the hybrid key issue, PR bitcoin#5247 disallowed any unparsable keys in
multisigs, whether or not they were hybrid, and whether or not the signature
was meant to satify a hybrid key.

In production Counterparty UTXOs were inadvertantly caught up in this
standardness rule change and became "soft confiscated", that is not longer
spendable by policy but still recoverable if users are able to somehow bypass
standardness by mining their transactions themselves, or getting out-of-band
assistance from some other miner.

I understand that Bitcoin Core never intended to "soft confiscate" any UTXOs by
policy changes.  This change addresses the problem.

With this change standardness rules intended for hybrid keys are only checked
after passing a signature check in CHECKMULTISIG operations.  Failing signature
checks revert to their consensus behaviour of testing subsequent public keys.

Counterparty UTXOs were never intended to make use of hybrid keys, and thus
shouldn't have any passing signatures using hybrid keys.

Author: roconnor-blockstream

src/script/interpreter.cpp

@@ -1158,7 +1158,7 @@ bool EvalScript(std::vector<std::vector<unsigned char> >& stack, const CScript&
                         // Note how this makes the exact order of pubkey/signature evaluation
                         // distinguishable by CHECKMULTISIG NOT if the STRICTENC flag is set.
                         // See the script_(in)valid tests for details.
-                        if (!CheckSignatureEncoding(vchSig, flags, serror) || !CheckPubKeyEncoding(vchPubKey, flags, sigversion, serror)) {
+                        if (!CheckSignatureEncoding(vchSig, flags, serror)) {
                             // serror is set
                             return false;
                         }
@@ -1167,6 +1167,10 @@ bool EvalScript(std::vector<std::vector<unsigned char> >& stack, const CScript&
                         bool fOk = checker.CheckECDSASignature(vchSig, vchPubKey, scriptCode, sigversion);
 
                         if (fOk) {
+                            if (!CheckPubKeyEncoding(vchPubKey, flags, sigversion, serror)) {
+                                // serror is set
+                                return false;
+                            }
                             isig++;
                             nSigsCount--;
                         }