rubygems: Replace OpenSSL::PKey::{RSA,DSA,EC}#to_pem and remove #private?

[junaruga]

What was the end-user or developer problem that led to this PR?

The current code OpenSSL::PKey::{RSA,DSA,EC}#to_pem and #private? makes the code complicated when adding ML-DSA on the PR #9697. OpenSSL::PKey::{RSA,DSA,EC}#to_pem is also not encouraged to use according to the Ruby OpenSSL documents.

• https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_rsa.c#L223-L280
• https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_dsa.c#L213-L271
• https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_ec.c#L402-L459

What is your fix for the problem, implemented in this PR?

Remove OpenSSL::PKey::{RSA,DSA,EC}#to_pem and #private?.

Summary

This PR has the following 5 commits to remove all of OpenSSL::PKey::{RSA,DSA,EC}#to_pem and #private? entirely in this repository.

1. Split Gem::Security.write into write_private_key and write_certificate
2. Replace OpenSSL::PKey::{RSA,DSA,EC}#to_pem
3. Recreated testing .pem files.
4. Remove hardcoded key-dependent values from tests
5. Remove #private? check from open_private_key and check in Gem::Security.sign

The 1st commit is a preparation for the 2nd commit to remove #to_pem. The 3rd commit is affected by the 2nd commit. The 3rd commit is to recreate testing .pem files as I changed the private key output format by the 2nd commit. I will explain the details on later section. The 4th commit is affected by 3rd commit. The 4th commits fixes the tests that broke when I recreated the testing .pem files. The 5th commit is to remove #private?.

Commit messages

1st commit

Split Gem::Security.write into write_private_key and write_certificate

Gem::Security.write managed the following 2 cases.

• pemmable is RSA/DSA/EC private key, an instance of OpenSSL::PKey::{RSA,DSA,EC}
• pemmable is certificate, an instance of OpenSSL::X509::Certificate

There was no case that pemmable was public RSA/DSA/EC key, an instance of
OpenSSL::PKey::{RSA,DSA,EC}.

This situation is not convenient in the case that pemmable is ML-DSA
private key, an instance of OpenSSL::PKey::PKey which doesn't have #to_pem, and instead
has #public_to_pem and #private_to_pem as alternative.

The Gem::Security.write with optional passphrase and cipher in certificate case
is not intuitive.

So, refactored the code as follows:

• Add Gem::Security.write_private_key for writing private keys, with optional
  passphrase and cipher for encryption
• Add Gem::Security.write_certificate for writing certificates, without
  passphrase and cipher. Certificates are never encrypted.
  OpenSSL::X509::Certificate#to_pem doesn't have passphrase/cipher arguments
  https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_x509cert.c#L169
• Remove Gem::Security.write
• Update all callers to use the new methods
• Rename test methods to test Gem::Security.write_private_key and add
  test_class_write_certificate

Assisted-by: Claude:claude-opus-4-6[1m]

2nd commit

Replace OpenSSL::PKey::{RSA,DSA,EC}#to_pem

Replace OpenSSL::PKey::{RSA,DSA,EC}#to_pem with
OpenSSL::PKey::{RSA,DSA,EC}#private_to_pem or #public_to_pem.
Continue to use OpenSSL::X509::Certificate#to_pem.

This makes the code ML-DSA compatible since OpenSSL::PKey::PKey used for ML-DSA
doesn't have #to_pem but has #private_to_pem and #public_to_pem.

Note this changes the private key's output format from the following formats to
more common PKCS #8 format.[1][2][3][4]

• RSA: PKCS Suggest similar gem names when install fails to find any matches #1 RSAPrivateKey
• DSA: traditional OpenSSL DSAPrivateKey
• EC: SEC 1/RFC 5915 ECPrivateKey

OpenSSL::PKey.read has supported PKCS #8 format as well as previous RSA/DSA/EC
formats since the beginning.

OpenSSL::PKey::{RSA,DSA,EC}#to_pem are not encouraged to use.[1][2][3]
The public key is format is unchanged.[1][2][3][5]

[1] OpenSSL::PKey::RSA#to_pem
https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_rsa.c#L223-L280
[2] OpenSSL::PKey::DSA#to_pem
https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_dsa.c#L213-L271
[3] OpenSSL::PKey::EC#to_pem
https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey_ec.c#L402-L459
[4] OpenSSL::PKey::PKey#private_to_pem
https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey.c#L991-L1010
[5] OpenSSL::PKey::PKey#public_to_pem
https://github.com/ruby/openssl/blob/27c59a4cc8a652004184182f734de0161ba08b41/ext/openssl/ossl_pkey.c#L1082-L1093

Assisted-by: Claude:claude-opus-4-6[1m]

3rd commit

Recreated testing .pem files.

As I modified logic related to output format in tool/create_certs.rb and
tool/create_encrypted_key.rb at the last commit, I ran these scripts to
recreate the testing .pem files as follows. Now the private .pem files are
PKCS #8 format.

$ ruby tool/create_certs.rb
$ ruby tool/create_encrypted_key.rb

4th commit

Remove hardcoded key-dependent values from tests

Because these hardcoded values make the tests fail when recreating the keys.
This is not convenient.

• Remove hardcoded subjectKeyIdentifier value assertions.
• In test_sign, replace assertion with hardcoded value with
  PRIVATE_KEY.verify (OpenSSL::PKey::RSA#verify) to assert
  signature

Assisted-by: Claude:claude-opus-4-6[1m]

5th commit

Remove #private? check from open_private_key and check in Gem::Security.sign

Remove the OpenSSL::PKey::{RSA,DSA,EC}#private? check in open_private_key. This
makes the code ML-DSA compatible since OpenSSL::PKey::PKey used for ML-DSA
doesn't have #private?.

According to the discussion on ruby/openssl#1085,
there is no reliable way to check if key is private or not in
OpenSSL::PKey::PKey for now. We shouldn't use
OpenSSL::PKey::{RSA,DSA,EC}#private?.

If a public key is passed, the subsequent signing operations will raise
ArgumentError in Gem::Security.sign. Rescue the ArgumentError, and raise
Gem::Security::Exception in Gem::Security.sign.

Add the following 2 bad key tests.

• test_execute_build_bad_key to test --build with a public
  key (bad key) as --private-key
• test_execute_sign_bad_key to test --sign with a public
  key (bad key) as --private-key

Assisted-by: Claude:claude-opus-4-6[1m]

Scripts to check private key output formats related to the 2nd commit

I prepared scripts for you to check changed private key output formats if you want to do it.

test_rsa.rb

require 'rubygems/security'
key = Gem::Security.create_key "rsa"
Gem::Security.write_private_key key, "private_key_rsa.pem"
Gem::Security.write_private_key key, "private_key_rsa.pem", 0o600, "pass"

test_dsa.rb

require 'rubygems/security'
key = Gem::Security.create_key "dsa"
Gem::Security.write_private_key key, "private_key_dsa.pem"
Gem::Security.write_private_key key, "private_key_dsa.pem", 0o600, "pass"

test_ec.rb

require 'rubygems/security'
key = Gem::Security.create_key "ec"
Gem::Security.write_private_key key, "private_key_ec.pem"
Gem::Security.write_private_key key, "private_key_ec_pass.pem", 0o600, "pass"

You can run the scripts as follows.

$ ruby -I lib test_rsa.rb
$ ruby -I lib test_dsa.rb
$ ruby -I lib test_ec.rb

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[junaruga]

@hsbt and @rhenium I would appreciate your reviews. Thank you.