8271566: DSA signature length value is not accurate in P11Signature

[zzambers]

Backport to fix failure we are seeing in some of our tests for jdk11 in FIPS mode (using sunpkcs11 provider):

java.security.ProviderException: cancel failed
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.cancelOperation(P11Signature.java:332)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.reset(P11Signature.java:272)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.engineVerify(P11Signature.java:755)
	at java.base/java.security.Signature$Delegate.engineVerify(Signature.java:1416)
	at java.base/java.security.Signature.verify(Signature.java:790)
...
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ARGUMENTS_BAD
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_VerifyFinal(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11Signature.cancelOperation(P11Signature.java:305)
	... 14 more

Backport only affects PKCS11 provider. Not clean because JDK-8242332 (missing in jdk11), made small changes (in-between) to engineSign and cancelOperation methods in P11Signature class in regards to DSA, which were then completely overwritten by this change set (so not an actual dependency).

Testing:
Fixes issue higher.
GHA: OK
pkcs11 tests (test/jdk/sun/security/pkcs11): OK
ssl-tests with TEST_PKCS11_FIPS=1: OK

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8271566 needs maintainer approval

Issue

• JDK-8271566: DSA signature length value is not accurate in P11Signature (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev.git pull/3166/head:pull/3166
$ git checkout pull/3166

Update a local copy of the PR:
$ git checkout pull/3166
$ git pull https://git.openjdk.org/jdk11u-dev.git pull/3166/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3166

View PR using the GUI difftool:
$ git pr show -t 3166

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/3166.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back zzambers! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (2e2c2e11)

[gnu-andrew]

I seem to remember there were reasons for not backporting this before. Given the follow-on issues, it should certainly wait until we start the July cycle next week.

@franferrax can you give your take on the risk of this one? Thanks.

[bridgekeeper]

@zzambers This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

[zzambers]

/touch

[openjdk]

@zzambers The pull request is being re-evaluated and the inactivity timeout has been reset.