CLOUDSTACK-9593: userdata: enforce data is a multiple of 4 characters

[marcaurele]

Python base64 requires that the string is a multiple of 4 characters but
the Apache codec does not. RFC says is not mandatory but the data should
not fail the VR script (vmdata.py).

Signed-off-by: Marc-Aurèle Brothier m@brothier.org

[wido]

LGTM

[jburwell]

@marcaurele this change looks a good check to add to LTS to as well. Could you please change the base branch to 4.9? Once you do, I will kick regression tests across all hypervisors in order to merge the fix.

[marcaurele]

@jburwell done

[marcaurele]

I changed the code to make to use the validation of the base64 value which is decoded to return the encoded one. This will happen any necessary padding to the value.
Fixing the incorrect user data in the database is not as straightforward as I thought.

[yadvr]

@marcaurele can you check why the Travis run failed

[marcaurele]

@rhtyd the SQL function I'm using to fix current user data in the database is not present in MySQL 5.5, but only in 5.6 (TO_BASE64, FROM_BASE64). I have to find a workaround, either in SQL or in Java to fix previous data.

[marcaurele]

@rhtyd The DB update class for the DatabaseUpgradeChecker doesn't work on Travis. Is there something I'm doing wrong, or missed something?

[yadvr]

@marcaurele an upgrade path such as Upgrade4910to4920 would only work if your destination/final version is 4.9.2.0, have a look at the db upgrade checker workflow and you'll see how to trigger your changes.

[marcaurele]

right, I forgot about the match required with the version in the pom. I thought I should prepare the stuff for the next 4.9.x release but that cannot work unless the pom files are updated too. I will just move the code inside Upgrade490to4910.java to Upgrade4910to4920.java when it's available.

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-829

[yadvr]

@marcaurele due to db changes in older upgrade paths, this cannot be accepted in 4.9/4.10, please re-base and change base branch to master.

[marcaurele]

@rhtyd I still don't get it why on new release there isn't the creation of the SQL migration files done by the RM. It's totally unclear where/how I should push my changes for the SQL upgrade.

[yadvr]

@marcaurele we already have 4.10.0.0 release and branch cut, we won't be able to accept PRs with schema changes for 4.9 branch since 4.9.2->4.9.3 path ought not change as it's included in 4.10.0.0 release. As I suggested earlier, change the base branch of the PR to master and put schema changes in 4.10->4.11 upgrade path.

[yadvr]

@marcaurele I checked there exists a 4.10.0.0 -> 4.11.0.0 upgrade path on master branch that you can use.

[marcaurele]

@rhtyd should be all set now

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[marcaurele]

@rhtyd conflict fixed

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1425

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1828)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28907 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1760-t1828-kvm-centos7.zip
Smoke tests completed. 62 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_vpc_privategw_acl	Failure	46.70	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Failure	218.40	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	Failure	198.33	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	Failure	308.92	test_privategw_acl.py
test_02_create_template_with_checksum_sha1	Error	5.26	test_templates.py
test_03_create_template_with_checksum_sha256	Error	5.22	test_templates.py
test_04_create_template_with_checksum_md5	Error	5.24	test_templates.py
test_08_resize_volume	Error	312.38	test_volumes.py
test_09_delete_detached_volume	Error	186.87	test_volumes.py
test_10_list_volumes	Error	172.32	test_volumes.py
test_01_vpc_remote_access_vpn	Error	45.81	test_vpc_vpn.py
test_hostha_kvm_host_degraded	Error	26.01	test_hostha_kvm.py

[yadvr]

@marcaurele can you check volume related failures, maybe caused by your changes?

[marcaurele]

@rhtyd failure due to VM not able to start/restart due to insufficient capacity. Are those tests run in parallel on the integration server?

[yadvr]

@marcaurele no they are not run in parallel, but several times when there is an exception/issue on the mgmt server side this general error is thrown. I'm suspecting if this has to do with any exception (NPE etc?) when VM starts? I can check again later today, meanwhile see if you find any issue or think that tests failures were env caused.

[yadvr]

Let's do a final test run and then this should be good to merge based on results
@blueorangutan package

[yadvr]

Fixed conflict. Re-testing...
@blueorangutan package

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-1486

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1917)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34746 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1760-t1917-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 64 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_network_garbage_collector_nics	Failure	301.34	test_vpc_redundant.py
test_hostha_enable_ha_when_host_in_maintenance	Error	0.83	test_hostha_kvm.py

[yadvr]

Merging this based on two lgtms and test results (ignoring env related Trillian failures).