Fix i2d_ASN1_TYPE() error check

[ndossche]

The OpenSSL docs / LibreSSL docs are slightly misleading wrt the error return values. 0 is also an error value.
This can be proven by looking at the call chain:
i2d_ASN1_TYPE -> ASN1_item_i2d -> asn1_item_flags_i2d -> ASN1_item_ex_i2d and noticing that ASN1_item_ex_i2d can return 0 on error.

This was found by a hybrid static-dynamic analyser that looks for inconsistent handling of error checks in bindings.

[rhenium]

The inconsistent return values look like more of a bug in OpenSSL which should be fixed there: https://github.com/openssl/openssl/blob/5971d32cfbf6ee34b0b4681ed693baddfe3573b4/crypto/asn1/tasn_enc.c

If this isn't causing an actual issue in ruby/openssl, I prefer not to introduce a workaround.

[ndossche]

That's fair, although the extension already works around it in the OCSP code:

openssl/ext/openssl/ossl_ocsp.c

Lines 448 to 449 in 1c4bfef

	if((len = i2d_OCSP_REQUEST(req, NULL)) <= 0)
	ossl_raise(eOCSPError, NULL);